Head-mounted display and projection screen

ABSTRACT

A head-mounted display comprises an image source configured to output one or more image components and one or more optical element configured to receive the one or more image components and output one or more images onto a projection screen.

RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 16/719,824, filed Dec. 18, 2019, which is acontinuation of U.S. patent application Ser. No. 16/565,328 filed Sep.9, 2019, which is a continuation-in-part application of U.S. patentapplication Ser. No. 16/370,694, filed Mar. 29, 2019, which isincorporated by reference herein.

TECHNICAL FIELD

The systems, apparatuses and methods described herein generally relateto video projection systems and, in particular, to video projectionsystems for near-eye displays, such as in virtual reality headsets.

BACKGROUND

Since the early days of computing and television, display systems haverelied on displaying of visual information across a screen. Through theyears, processing power and miniaturization have allowed the screenresolution to increase dramatically, but the basic approach of uniformlydisplaying pixels across the screen has prevailed. However, thisapproach requires significant increases in communications andcomputational performance to deliver all of the pixels as the resolutionincreases. These problems have become particularly acute with the adventof virtual reality headsets, where the images, when viewed through, butnot limited to, an eyepiece or waveguide cover significant amount of theviewer's field of view compared to traditional displays and end uphaving some of their pixels usually or always in or near to the viewer'speripheral vision.

Traditional displays have pixels or scanlines with fixed sizes anddistances from each other in typically a regular grid or similaruniformly distributed pixel or scanline pattern on a flat or slightlycurved screen. See FIG. 1A which shows the single pixel 101 approach todisplay devices such as LCD (Liquid crystal display) or OLED (Organiclight-emitting diode) computer or television displays. FIG. 1B shows thescanline approach 102 used in other display devices such as CRT(Cathode-ray tube) computer or television displays and CRT or LBS (Laserbeam steering) video projectors. But the eye interprets the field ofvision 103 with high resolution at the center 104 and a decreased visionat the periphery 105, as seen in FIG. 1C. Although human vision is quitedifferent from the single pixel 101 or scanline 102 design with far morephotoreceptor cells and visual acuity in the foveal vision 104, thiskind of fixed and even distribution of pixels or scanlines ensures asimilar quality image when viewing every part of a screen from manydistances and angles.

Current examples where this uniform distribution of pixels or scanlinesdoes not apply is very limited and mostly unintentional, for example inthe projection mapping industry where often 3d surfaces are used as thescreens of video projectors.

Lately, a need for variable-resolution screens 103 has emerged becauseof increasing manufacturing costs of high resolution microdisplays,displays and projectors and much more demanding computational, bandwidthand storage requirements for display content created for traditionalscreens due to their increasing resolutions and fields of view,especially in virtual reality, augmented reality and mixed realityheadsets (from now on referred to as “XR headsets”).

Current XR headsets aim to provide a field of view close to the humanfield of view, which is on average 270 degrees horizontally by 135degrees vertically taking into account eye rotations and is usuallylower than that, for example 90 degrees horizontally by 100 degreesvertically for virtual reality headsets and lower than 50 degreeshorizontally by 50 degrees vertically for augmented reality headsetswhich is still higher than many screens at normal viewing distances suchas monitors, TVs and projection screens.

Other examples are video projectors that can be set up to project verywide and cover more of the viewer's field of view than with displaytechnologies such as CRT, LCD, OLED or microLED monitors and TVs andprojection screens at normal viewing distances.

A hybrid of the two is also a potential use case for this method anddisplay apparatus such as has been demonstrated by HMPDs (Head-MountedProjective Display) which are both a head-mounted device but projectonto a retroreflective projection screen like the ones used for videoprojectors rather than to a waveguide or projection screen viewed withan eyepiece lens or other optics similar to other XR headsets.

At such high fields of view, the same amount of pixels or scanlinesprovides less pixels or scanlines per degree of the field of view of theviewer and can suffer from noticeable lack of detail, pixelation andscreen-door effect or gap between scanlines.

Current methods of displaying less pixels in the periphery is done byhaving very high pixel density everywhere on the display and displayingless resolution on the pixels displayed near or in the viewer'speripheral vision rather than having less pixels or scanlines there tobegin with. This is a technique the Sony PlayStation VR and Oculus Gohead-mounted displays use (similar to 103).

This approach of increasing the pixel or scanline count on the displayuniformly poses both cost and computational challenges as way morepixels or scanlines are required to cover the high fields of view,especially for the average human field of view of 270 degreeshorizontally (195 degrees per eye) by 135 degrees vertically which for a60 pixels per degree resolution needed for a 20/20 vision would requireabout 11,700 pixels horizontally and 8100 pixels vertically per eye.

Manufacturing custom screens with more pixels where the viewer's fovealview can reach will be very expensive and require custom displaycontrollers.

Even if it were possible and economically feasible, the computationalpower required for creating real-time foveated content described abovefor such screens could be used for other tasks such as rendering anddisplaying more detailed virtual reality images in real-time.

So far methods have been proposed of optically combining two projectorsor displays to achieve variable-resolution screens such as with a beamsplitter. There are disadvantages to this approach such as higher cost,weight, size, requirement for color correction and synchronizationbetween different displays or projectors and only being able to have onehigh resolution part and one low resolution part on the image with twodisplays or projectors (see the teachings in the following patents:US20160240013A1, U.S. Pat. Nos. 9,711,072B1, 9,983,413B1, 9,989,774B1,9,711,114B1, 9,905,143B1).

Also, tilting beam splitters or steering an image with mirrors or prismsto reposition the high resolution area is challenging and results inperspective distortion and some optical aberrations which some of themethods described herein solve. Additionally, tilting or rotatingmechanical parts have disadvantages associated with mechanically movingparts which some of the methods described herein solve.

BRIEF SUMMARY OF THE INVENTION

An optical apparatus for creating a variable-resolution image stream ona screen is described herein that is made up of a projector connected toa video source, where the projector transmits a light image stream inthe form of a high resolution, small image component and a lowresolution, large image component. Each frame of the variable-resolutionimage stream may be or include one of a) a low resolution, large image,b) a high resolution, small image, or a superimposition of a highresolution, small image and a low resolution, large image. This lightimage stream is sent to an image steering element that directs the highresolution, small image component and the low resolution, large imagecomponent to a small image optical element and to a large image opticalelement. Additionally, the image steering element may function as animage separation element, and may separate the first image componentfrom the second image component in embodiments. The optical apparatusmay also include an image separation element that separates the highresolution, small image component and the low resolution, large imagecomponent into a high resolution, small image stream and a lowresolution, large image stream, where the small image optical elementand the large image optical element focus the low resolution, largeimage stream and the high resolution, small image stream on the screensuch that the low resolution, large image stream and the highresolution, small image stream appear as the variable-resolution imagestream on the screen.

In some embodiments, the light image stream from the projector is timemultiplexed between the high resolution, small image component in afirst frame (frame n) and the low resolution, large image component in anext frame (frame n+1). The image separation element could be an opticalshutter to manage the time multiplexing. Alternately, the light imagestream from the projector could have the high resolution, small imagecomponent on one part of each image and a low resolution, large imagecomponent on another part of the image. The image separation elementcould be an optical mask (stencil) to support this embodiment.

In some embodiments, the screen is embedded in a virtual realityheadset. The small image optical element could include a lens array. Theimage steering element could be a rotating optical slab, mirrors, beamsplitter (e.g., polarizer beam splitter or reflective polarizer beamsplitter), wedge (Risley) prisms, liquid crystal switchable mirrors,optical shutters or optical masking elements. The large image opticalelement could be a lens or other optics that focuses the low resolution,large image stream to an outer portion of the screen or viewer's fieldof view. The small image optical element could be a lens or other opticsthat focuses the high resolution, small image stream to a center portionof the screen or viewer's field of view.

An optical method creating a variable-resolution image stream on ascreen is described herein, where the method includes the steps ofcreating a light image stream in the form of a high resolution, smallimage component and a low resolution, large image component with aprojector connected to a video source; directing the high resolution,small image component and the low resolution, large image component,with an image or beam steering element, to a small image optical elementand to a large image optical element; separating the high resolution,small image component and the low resolution, large image component intoa high resolution, small image stream and a low resolution, large imagestream with an image separation element; and focusing, by the smallimage optical element and the large image optical element, the lowresolution, large image stream and the high resolution, small imagestream to form the variable-resolution image stream on the screen.

In some embodiments of the optical method, the light image stream fromthe projector is time multiplexed between the high resolution, smallimage component in a first frame (frame n) and the low resolution, largeimage component in a second frame (frame n+1). The separation of thesecomponents could be accomplished by using an optical shutter for theimage separation element. In another embodiment of the optical method,the light image stream from the projector could have the highresolution, small image component on one part of each image and a lowresolution, large image component on another part of the image, and theimage separation element could be an optical mask (stencil).

In some embodiments of the optical method, the screen is embedded in avirtual reality headset. The small image optical element could include alens array. The image steering element could be a rotating optical slab,mirrors, beam splitter, wedge (Risley) prisms, optical shutters, liquidcrystal switchable mirrors, or optical masking elements. The large imageoptical element could be a lens or other optics that focuses the lowresolution, large image stream to an outer portion of the screen orviewer's field of view. The small image optical element could be a lensor other optics that focuses the high resolution, small image stream toa center portion of the screen or viewer's field of view. The screencould be a flat or curved diffuse projection screen, a flat or curvedretroreflective projection screen, a flat or curved holographic diffuserprojection screen, a flat or curved fiber optic taper bonded to a firstsurface or projection screen, or a flat or curved mirror or Fresnelmirror which focuses a projection onto a viewer's retina (such as theones used in collimated display systems). The screen could also be aviewer's retina. The projector could be a microdisplay or a display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows the single pixel approach to display devices.

FIG. 1B shows the scanline approach to display devices.

FIG. 1C shows a multi-focus approach to a display.

FIG. 2 illustrates using persistence of vision to blend images from twoconsecutive frames into one final image.

FIG. 3 shows splitting an image of a microdisplay, display or projectorinto two parts that are combined.

FIG. 4A is a functional flow of the light through the optical functionsin the simplest embodiment.

FIG. 4B is a hardware flow of the light through the optical elements inthe simplest embodiment.

FIG. 5A is a functional flow of the light through the optical functionsin a slightly more complex embodiment.

FIG. 5B is a hardware flow of the light through the optical elements ina slightly more complex embodiment.

FIG. 5C is a hardware flow of the light through the optical elements asin the previous drawing with an optical mask (stencil).

FIG. 5D is a hardware flow of the light through the optical elements asin the FIG. 5B for each eye in a head-mounted display.

FIG. 5E illustrates an embodiment using optical slab elements.

FIG. 5F is a hardware flow of the light through the optical elements ina slightly more complex embodiment, using a screen.

FIG. 5G is a hardware flow of the light through the optical elements asin the previous drawing with an optical mask, using a screen.

FIG. 5H is a hardware flow of the light through the optical elements asin the FIG. 5F for each screen in a head-mounted display.

FIG. 6A shows an illustration with rectangles representing individualpixels.

FIG. 6B shows an illustration with individual pixels displaying anactual image.

FIG. 7A shows an original image.

FIG. 7B shows a perspective distorted image.

FIG. 8A shows the image with no distortion or distortion mismatchcorrected.

FIG. 8B shows the image with the distortion mismatch.

FIG. 9A shows light going through an optical slab.

FIG. 9B shows light going through an optical slab at an angle.

FIG. 9C shows light going through an optical slab at a different angle.

FIG. 9D shows a superimposition of light going through the same opticalslab at two opposite maximum angles.

FIG. 10A illustrates offsetting the image or beam with two mirrorstilted 45 degrees.

FIG. 10B illustrates offsetting the image or beam with two mirrorstilted 40 degrees.

FIG. 11 illustrates offsetting the image or beam with a set of fourmirrors.

FIG. 12 illustrates offsetting the image or beam with a set of Doveprism and four mirrors.

FIG. 13 shows a functional flow of light through the optical functionsin a lens array embodiment.

FIG. 14 shows a functional flow of light through the optical functionsin a second lens array embodiment.

FIG. 15 shows the light flow through the elements of one lens arrayembodiment.

FIGS. 16A and 16B illustrates an image (FIG. 16A) as it is seen on ascreen or viewer's retina (FIG. 16B) after using a lens array.

FIG. 17A shows an embodiment using a matched set of lens arrays and anoptical masking element.

FIG. 17B demonstrates reflective microdisplay used for the opticalmasking element.

FIG. 17C shows the use of a display, such as an LCD display with itsreflective and backlight layers removed as the optical masking element.

FIG. 17D illustrates the elements for generating an image with large andsmall parts that are already optically combined, with the ability tohide the duplicated images of the small part and show one of them.

FIG. 17E shows the elements for generating an image with large and smallparts that are already optically combined, with the ability to hide theduplicated images of the small part and show one of them.

FIG. 17F demonstrates the elements for another embodiments forgenerating an image with large and small parts that are alreadyoptically combined, with the ability to hide the duplicated images ofthe small part and show one of them.

FIG. 17G presents another embodiments of the elements for generating animage with large and small parts that are already optically combined,with the ability to hide the duplicated images of the small part andshow one of them.

FIG. 18 illustrates digitally and optically rearranging portions of animage.

FIG. 19 shows the portions of the images to be optically and digitallyrearranged do not have to be partitioned from the middle of the images.

FIG. 20 shows an embodiment of the mounting of the present inventions ina head-mounted display where the physical size is reduced.

FIG. 21 shows an embodiment of the mounting of the present inventions ina head-mounted display using a mirror to reduce the physical size of theunit.

FIG. 22 shows an embodiment of the mounting of the present inventions ina head-mounted display using a mirror and a beam splitter to reduce thephysical size of the unit.

FIG. 23 illustrates an embodiment that uses a combination of a firstoptical source that outputs an image onto a screen to show a firstportion of a variable resolution image and second optical source to showa second portion of the variable resolution image.

FIG. 24A illustrates a source image, in accordance with an embodiment.

FIG. 24B illustrates the source image of FIG. 24A on an optical maskingelement that is on an intermediate image plane, in accordance with anembodiment.

FIG. 24C illustrates the source image of FIG. 24A on an optical maskingelement that is slightly offset from the intermediate image plane, inaccordance with an embodiment.

FIG. 25 shows how spatially modulating an image twice or spatiallymodulating two images which store different bit depth (color depth)information of an original image, from left to right, produces a highercontrast and/or higher bit depth (color depth) image on the right.

FIG. 26 shows, from left to right, how spatially modulating a highresolution, small image again or spatially modulating a different bitdepth (color depth) information of the high resolution, small image onthe optical masking element, where it is displayed at a lowerresolution, produces a higher contrast and/or higher bit depth (colordepth) image on the right.

FIGS. 27A-27D illustrate how non-mechanical elements may shift an imagebeam and position an image, in accordance with an embodiment.

FIGS. 28A-28C illustrate various non-mechanical beam steering elementsand how the beam may be shifted with each, in accordance with anembodiment.

FIG. 29 illustrates a fiber optic taper or faceplate from four differentangles, in accordance with an embodiment.

FIG. 30 illustrates how a fast refresh rate image source may be usedwith a low resolution high refresh rate optical masking element forpositioning of the small image on the final image.

FIG. 31 illustrates various methods of increasing the resolution of animage, in accordance with an embodiment.

FIG. 32A illustrates how a curved projection screen with engineeredreflective and scattering properties may be used to both steer theprojection beam as well as scatter it by a specific amount to achieve avery high gain projection screen, in accordance with an embodiment.

FIG. 32B illustrates how a curved mirror, beam splitter and rearprojection screen with engineered scattering properties may be used tosteer the projection beam by the curved mirror and scatter it by aspecific amount by the rear projection screen to achieve a very highgain projection screen, in accordance with an embodiment.

FIG. 32C illustrates a different configuration of how a curved mirror,beam splitter and rear projection screen with engineered scatteringproperties may be used to steer the projection beam by the curved mirrorand scatter it by a specific amount by the rear projection screen toachieve a very high gain projection screen, in accordance with anembodiment.

FIG. 33 illustrates how the pixel information displayed by the highresolution small image and the region of the low resolution large imagethat corresponds to the high resolution small image may be used togetherto increase at least one of a contrast or bit depth (color depth) of theregion of the final image corresponding to the high resolution smallimage, in accordance with an embodiment.

FIG. 34 illustrates various visual effects that may be applied beyond asource 2D frame on a variable resolution screen, in accordance with anembodiment.

DETAILED DESCRIPTION

The present inventions describe a system and method for implementing avariable-resolution screen, where the area in front of the viewer'sfield of view, where the foveal vision expects the greatest resolution,are in a higher resolution than the areas of the screen on theperiphery, where the peripheral vision expects less resolution andclarity. In this application four major (and many minor) embodiments aredescribed.

The following inventions describe a method and display apparatus forachieving a variable-resolution screen, which can be defined as a screenwhich allows the image, when viewed directly or by, but not limited to,an eyepiece (the lens closest to the viewer's eye) or waveguide, providea resolution which is not uniform across the image but rather morepixels or scanlines are visible to the viewer where needed on the image,such as the center of the viewer's field of view and less in other partor parts of the image.

Such a screen is different from existing screens displaying pre-renderedor real time-rendered foveated content as such methods ofvariable-resolution content display limit the high resolution part ofthe content to the native resolution possible with that part of thescreen. The term “screen” can also be used to describe the viewer'sretina.

Foveated content is an image, video or real time-generated images whereon each image the resolution varies across the image, for example toshow more resolution only where the viewer is looking, is able to lookat or is meant to look at.

The variable-resolution screen methods and apparatus described hereallow to achieve more resolution visible in one or more parts of theimage than is possible with the microdisplay, display or projector whenused without the methods described here.

The methods described may be performed using existing computing hardwaresuch as a PC, mobile phone or tablet to provide the pre-rendered or realtime-rendered content for it.

The methods may be performed with as little as a single DLP (Digitallight processing), LCoS (Liquid crystal on silicon), LCD (Liquid crystaldisplay), OLED (Organic light-emitting diode), MicroLED or similarmicrodisplay, display or projector or LBS (Laser beam steering) orsimilar projector 401, 411, 501, 511, 521, 551, 2301, 5111, 5121, 5151,1401 for one variable-resolution screen or one of the above for onevariable-resolution screen per eye, for example for head-mounteddisplays. Using as little as a single microdisplay, display or projectoror one per eye allows to minimize the cost of producing such avariable-resolution screen apparatus, reduce weight and size of theapparatus. A single microdisplay, display or projector can also refer tomicrodisplays, displays or projectors where a separate display ormicrodisplay panel is used for each color channel and they are opticallycombined such as with a trichroic prism, X-cube prism or dichroicfilters. This can be useful for various reasons such as eliminatingcolor separation (also known as “rainbow artifact”) and increasing therefresh rate.

The usage of such variable-resolution screens are, but not limited to,virtual reality, augmented reality and mixed reality headsets (“XRheadsets”) and video projectors.

Positioning with Mirrors or Wedge Prisms of a High Resolution SmallImage Over a Low Resolution Large Image

In one embodiment, a variable-resolution screen can be achieved bypositioning a high resolution small image over a low resolution largeimage with mirrors or wedge (Risley) prisms.

To achieve a variable-resolution screen a single display technology suchas a microdisplay or display 401, 411, 501, 511, 521, 551 is operated atfast refresh rates. Each consecutive frame (frame n+1) the microdisplayor display is used to either display a small high resolution part 204 orparts of the final image 205 or a large low resolution part 203 or partsof the final image 205 by sharing the refresh rate of the frames 201,202 and final image 205 between the latter's two or more parts 203, 204.Persistence of vision blends the two parts 203, 204 into one final image205. See FIG. 2.

In FIG. 2, the frames alternate with the low resolution frame n 201displayed followed by high resolution frame n+1 202. With a sufficientrefresh rate, the eye interprets the two as a single image 205. The lowresolution portion of the combined screen 203 could have a neutral color(black) in the high resolution area 204. And the high resolution portionof the combined screen 204 could have a neutral color (black) in the lowresolution area 203. A slight overlap between the two regions 203, 204will prevent a noticeable seam or gaps by having a blend region whereregions 203, 204 overlap. In another embodiment, the low resolutionsection 203 is not masked and blends with the high resolution portion204 in the area where the high resolution resides.

Alternatively, to achieve a variable-resolution screen a single displaytechnology such as a microdisplay or display is optically split into twoor more parts 301, 302. This method allows one part 301 or parts to usemore pixels on the final image by sacrificing the resolution of anotherpart 302 or parts on the final image. See FIG. 3.

The two methods can also be combined to allow to create more parts onthe final image or to allow to create two or more final images bysharing both the resolution and refresh rate of the microdisplay ordisplay between the parts, such as for using a single microdisplay ordisplay to create final images for both eyes in a head-mounted display.

In FIG. 3, a 16:9 aspect ratio microdisplay or display split into twoparts 301, 302 is shown, for example 1920×1080 pixel microdisplay ordisplay split into a small 1080×1080 pixel high resolution part 301 anda large 840×1080 pixel low resolution part 302 (the latter may then beoptically flipped 90 degrees for a better aspect ratio).

Using optical or optical and mechanical and also optionally digitalmethods, the parts 301 and 302 can be resized and superimposed on eachother 305. The large low resolution part 303 can be masked where thesmall high resolution part is 304 and where they overlap.

The masking can further be made more seamless by blending the edgesoptically or digitally by making the transition less abrupt with adigital resolution falloff in the high resolution small image or dimmingthe pixels with a falloff on both images.

The brightness levels between the two parts may be balanced opticallysuch as with neutral density filters or digitally.

Look to FIGS. 4A and 4B. To be able to use the same microdisplay ordisplay 401, 411 for each part which have a different size and positionon the final image 405, with the first method from FIG. 2, the image ofthe microdisplay or display is steered with a steering elementoptomechanically or optically, such as, but not limited to, a rotatingmirror or beam splitter 402, 412 and an optional mirror 413, to one oftwo optical elements 403, 404, 414, 415 for each frame. Other examplesof image steering elements are a liquid crystal switchable mirror, anoptical shutter, a wedge prism, a rotating optical slab, and opticalmasking elements. The image steering element may also function as animage separation element. Alternatively, a separate image separationelement may be used in addition to the image steering element. Examplesof image separation elements include optical masking elements, a beamsplitter, an optical shutter, a liquid crystal switchable mirror, and soon. In case of using a beam splitter instead of a rotating mirror as thesteering element, each image each frame may be blocked or let to passaccordingly before, inside or after the optical element 403, 404, 414,415 with an optical or mechanical shutter such as an LCD shutter inorder to prevent 403, 414 and 404, 415 from receiving the same image ofevery frame instead of the different images of different consecutiveframes. This is of course not needed if a polarizer beam splitter or areflective polarizer beam splitter is used and the polarization of theimage can be controlled each frame before it reaches the beam splitter,such as with a switchable liquid crystal polarization rotator. Use of areflective polarizer beam splitter may provide improved image contrastand/or light throughput as compared to a half-silvered beam splitter oran absorptive-type polarizer beam splitter.

To be able to use the same microdisplay or display 401, 411 for eachpart which have a different size and position on the final image 405,with the second method from FIG. 3, the image of the microdisplay ordisplay 401, 411 is steered with a steering element such as, but notlimited to, a beam splitter, a mirror, or any of the otheraforementioned optical steering elements on an image plane 402, 412 andan optional mirror 413, to two optical elements 403, 404, 414, 415. Incase of using a beam splitter and not a mirror on an image plane, eachimage is then masked accordingly before, inside or after the opticalelement 403, 404, 414, 415 with an optical masking element such as astencil. The mirror or stencil may be on an image plane to create asharp cut.

Steering element 402, 412 may be, but is not limited to, a mirror,mirrors, beam splitter and optical or mechanical shutter or shutters(e.g., a liquid crystal switchable mirror) combined with one of theabove. The steering element 402, 412 may be configured to direct a firstimage component to a small image optical element and to direct a secondimage component to a large image optical element. In some embodiments,the large image optical element and the small image optical element arecompletely separate. In other embodiments, the small image opticalelement and large image optical element may share one or more of theirconstituents. For example the small image optical element and largeimage optical element may share most of their lenses, and the largeimage optical element may have extra lenses for making the beam widerwhich are unreachable to the narrow beam thanks to a reflectivepolarizer beam splitter. In some embodiments, a single optical elementfunctions as the large image optical element (or a component of thelarge image optical element) and the small image optical element (or acomponent of the small image optical element). For example, the singleoptical element may include one or more electrically tunable lens (e.g.,a liquid lens and/or a liquid crystal lens). An electrically tunablelens can change their focal length electrically, which means if properlyintegrated with other lenses, for time-sequential embodiments a singleoptical element can function as a large image optical element one frame,then as a small image optical element in the next frame. Thus, thesingle optical element can become at least a portion of the large imageoptical element at a first time and can become at least a portion of thesmall image optical element at a second time. Additionally, the steeringelement 402, 412 may function as an image separation element, and mayseparate the first image component from the second image component.

The optical element 403, 404, 414, 415 may be, but is not limited to,one of the following, or a combination of: lenses, mirrors, prisms,free-form mirrors.

One of the optical elements 404, 415 may create a small image 417 andthe other optical element 403, 414 a comparably large image 416.

In FIG. 4A, the microdisplay or display 401 creates the image andoptically sends it to the image or beam steering element 402. The imagesteering element 402 splits the image into two (or more) images, sendingthe images to optics creating the low resolution, large image 403 and ahigh resolution, small image 404. The optical output of the optics 403,404 are sent to a screen 418 or onto the viewer's retina where the finalimage is created 405.

Looking to FIG. 4B, the microdisplay or display 411 creates an imagethat is split with a beam splitter (such as half silvered mirror orpolarizer beam splitter) 412 into two identical images going indifferent directions. One is directed to optics which create a largeimage 414, while the other goes through a mirror 413 to another opticswhich creates a small image 415. The large image optics 414 create thelower resolution image 416. The small image optics 415 creates thehigher resolution image 417. Both the lower 416 and higher 417resolution images are projected on the screen 418 or on the viewer'sretina as seen in FIG. 5B.

Masking of the area of the large image 416 where the small image 417 iscan be achieved, again, digitally, by having black pixels displayedthere, or optically, for example by having a stencil on an image planesomewhere inside, before or after the optics to physically (optically)mask off that part of the image.

Then, optionally, the positioning of the small image can be achievedwith, but not limited to one or more of the following: actuators withmirrors, galvanometer scanners, actuators with wedge (Risley) prisms,actuators with tilting or shifting lenses, as seen in FIG. 5A.

FIG. 5A shows a microdisplay or display 501 creating an image that issent to an image or beam steering element 502 (could be a beamsplitter). One of the two identical images is sent to the large imageoptical element 503 and the other image is sent to the small imageoptical element 504. In some embodiments, the large image opticalelement 503 and the small image optical element 504 are completelyseparate. In other embodiments, the small image optical element 504 andlarge image optical element 503 may share one or more of theirconstituents. For example the small image optical element 504 and largeimage optical element 503 may share most of their lenses, and the largeimage optical element 503 may have extra lenses for making the beamwider which are unreachable to the narrow beam thanks to a reflectivepolarizer beam splitter. The small image optical element 504 sends theimage to an image or beam steering element 506 (could be a mirror). Theimages are then combined into a final image 505 (e.g., into asuperimposition of the two images).

In one embodiment, a single optical element functions as the large imageoptical element 503 (or as a component of the large image opticalelement 503) and the small image optical element 504 (or as a componentof the small image optical element 504). For example, the single opticalelement may include one or more electrically tunable lens (e.g., aliquid lens and/or a liquid crystal lens). An electrically tunable lenscan change its focal length electrically, which means if properlyintegrated with other lenses, for time-sequential embodiments a singleoptical element can function as large image optical element 503 oneframe, then as small image optical element 504 in the next frame.

The two images are optically combined, such as with a beam splitter andviewed directly, or through, but not limited to, an eyepiece orwaveguide. The optically combined images may be a superimposition of thetwo images.

Looking to FIG. 5B, the optical elements are shown. The microdisplay ordisplay 511 sends the image to a beam splitter or a rotating mirror 512that sends images to the large image optics 514 and to a mirror 513 thatredirects the image to the small image optics 515. From the small imageoptics 515, the image is sent to a mirror 519 and then to a beamcombiner 518 (e.g., beam splitter) to combine with the output of thelarge image optics 514. From the beam combiner 518, the large image 516and the small image 517, are sent as a combined image to the viewer'sretina 510 (e.g., a superimposition of the two images). In case of usinga beam splitter instead of a rotating mirror as the steering element,each image each frame may be blocked or let to pass accordingly before,inside or after the optical element 514, 515 with an optical ormechanical shutter such as an LCD shutter in order to prevent 514 and515 from receiving the same image of every frame instead of thedifferent images of different consecutive frames. This is of course notneeded if a polarizer beam splitter (e.g., a reflective polarizer beamsplitter) is used and the polarization of the image can be controlledeach frame before it reaches the beam splitter, such as with aswitchable liquid crystal polarization rotator.

One difference between FIGS. 5B and 5C is that images are illustrated astwo lines rather than one before reaching the optical masking elements.This is done to illustrate how the image is masked/cropped by theoptical masking elements 530, 531.

Looking to FIG. 5C, the optical elements are shown, for processing theimage structure in FIG. 3. The microdisplay or display 521 sends theimages to a beam splitter 522, that sends two identical images, one to amirror 523 first, to optical masking elements (stencils, physicalbarriers to hide part of the image) 530, 531. The stencil may be on animage plane to create a sharp cut, so can also be inside the optics (524and 525), or after the optics.

The images leave from the stencils 530, 531 to the large image optics524 and to the small image optics 525. From the small image optics 525,the image is sent to a mirror 529 and then to a beam combiner 528 (e.g.,beam splitter) to combine with the output of the large image optics 524.From the beam combiner 528, the large image 526 and the small image 527,are sent as a combined image to the viewer's retina 520.

Looking to FIG. 5D we see a head-mounted display embodiment which uses asingle microdisplay or display 551 for both eyes. First the resolutionof the microdisplay or display is split between eyes, then each framemay be used for one projection (large or small image). For example witha 240 Hz DLP microdisplay this provides 120 Hz refresh rate per imageper eye.

The microdisplay or display 551 sends the image to a beam splitter 560that sends two identical images, one to a mirror 580 first, to thestencils 561, 571 that mask off the portion of the image not destinedfor the specific eye. In one embodiment, the stencils 561, 571 could beshutters such as an LCD shutter or LCD pi-cell so each frame may be sentto one optics and blocked for the rest of the optics 554, 555, 574, 575,such as in the instance seen in FIG. 2. In another embodiment, thestencils 561, 571 could be removed so each frame the whole image may besent to one optics and blocked for the rest of the optics 554, 555, 574,575, such as in the instance seen in FIG. 2. For example with a 240 HzDLP microdisplay this provides 60 Hz refresh rate per image per eye.

The left stencil (top in the diagram) 561 sends the image to a secondbeam splitter 552 which send two identical images, one to a mirror 553first, to the two LCD shutters 562, 563 for the FIG. 2 embodiment. Theshutters 562, 563 could be replaced with stencils (a physical barrier tohide part of the image) for the FIG. 3 embodiment. The stencils have tobe on an image plane to create a sharp cut, so can also be inside theoptics (554 and 555), or after the optics.

The images leave from the shutters (or stencils) 562, 563 to the largeimage optics 554 and to the small image optics 555. From the small imageoptics 555, the image is sent to a mirror 559 and then to a beamcombiner 558 (e.g., beam splitter) to combine with the output of thelarge image optics 554. From the beam combiner 558, the large image 556and the small image 557, are sent as a combined image to the viewer'sretina 550.

The right stencil (bottom in the diagram) 571 sends the image to asecond beam splitter 572 which sends two identical images, one to amirror 573 first, to the two LCD shutters 580, 581 for the FIG. 2embodiment. The shutters 580, 581 could be replaced with stencils forthe FIG. 3 embodiment. The stencils may be on an image plane to create asharp cut, so can also be inside the optics (574 and 575), or after theoptics.

The images leave from the shutters (or stencils) 580, 581 to the largeimage optics 574 and to the small image optics 575. From the small imageoptics 575, the image is sent to a mirror 579 and then to a beamcombiner 578 (e.g., beam splitter) to combine with the output of thelarge image optics 574. From the beam combiner 578, the large image 576and the small image 577, are sent as a combined image to the viewer'sretina 570.

Due to persistence of vision with the method in FIG. 2 and masking withthe method in FIG. 3 the two parts appear as one uniform image 604 inFIG. 6B.

In FIG. 6A, the illustration shows rectangles representing individualpixels 601. FIG. 6B shows an illustration with individual pixelsdisplaying an actual image 604.

Since the small high resolution part 603, 606 in the final image 601,604 can be smaller than it could be without the use of these methods,the variable-resolution screen method and apparatus described hereallows to achieve more resolution visible in one or more parts of theimage than is possible with the display technology when used without themethods described here.

This allows to achieve a variable-resolution screen, such as ahead-mounted display screen which uses one microdisplay or display orone per eye with a high pixel or scanline density in the center of thefield of view of the viewer and less in the periphery.

Optionally, by adding eye tracking via, but not limited to, gazetracking cameras or electrodes, the small high resolution part 603, 606can be positioned on the final image 601, 604 on the large lowresolution part 602, 605 where the viewer's foveal view is at any givenpoint in time. This allows to always have more pixels or scanlinesconcentrated in the foveal and optionally also in the near peripheralview of the viewer at any given point in time.

Optionally the positioning of the large low resolution part 602, 605 canbe achieved the same way the positioning of the small high resolutionpart 603, 606, for example to have pixels only in the field of view ofthe viewer's eye and not the total field of view of the viewer whichtakes into account eye rotations.

There can also be more than two parts, such as three, one for the fovealview, one for near peripheral and one for far peripheral and they can becombined and optionally positioned the same way as mentioned above.

Those skilled in the art will understand that the order of some elementscan be changed and more can be added, such as steering both large andsmall images together after they are optically combined, or adding moreelements for creating more small or large parts on the final image.

Positioning with Mirrors or Wedge Prisms of a High Resolution NarrowProjection Beam Over a Low Resolution Wide Projection Beam

In another embodiment, a variable-resolution screen is achieved bypositioning a high resolution narrow video projection over a lowresolution wide video projection with mirrors or wedge (Risley) prisms.

To achieve a variable-resolution screen a single video projector such asa single illuminated microdisplay, display, LBS (Laser beam steering)projector or other type of video projector (from now on referred to as“projector”) 401, 411, 501, 511, 521, 551, 5111, 5121, 5151 is operatedat fast refresh rates. Each consecutive frame (frame n+1) the projectoris used to either display a small high resolution part 204 or parts ofthe final image 205 or a large low resolution part 203 or parts of thefinal image 205 by sharing the refresh rate of the frames 201, 202 andfinal image 205 between the latter's two or more parts 203, 204.Persistence of vision blends the two parts 203, 204 into one finalprojected image 205.

Alternatively, in FIG. 3, to achieve a variable-resolution screen 305 asingle video projector such as a single illuminated microdisplay,display, LBS (laser beam steering) projector or other type of videoprojector (from now on referred to as “projector”) is optically splitinto two or more parts 301, 302. This method allows one part 301, 304 orparts to use more pixels on the final projected image 305 by sacrificingthe resolution of another part 302, 303 or parts.

The two methods can also be combined to allow to create more parts onthe final projected image or to allow to create two or more finalprojected images by sharing both the resolution and refresh rate of theprojector between the parts, such as for using a single projector tocreate final projected images for both eyes in a head-mounted display.

There are several advantages to using projection beams rather thanmicrodisplays and displays when viewed directly or through lens or otheroptics:

First of all, it is very challenging to design a wide field of viewhead-mounted display when using microdisplays while trying to keep themagnification lenses or other optics small and lightweight versus usingmuch smaller projection lenses to project onto a screen larger than themicrodisplay and viewing that screen through lenses or other opticsinstead.

Second, using video projections has the advantage of allowing to haveall of the optical elements including steering elements be much smalleras they can be positioned in the optical design before, or somewhere inbetween the projection optics which create the large final image on aprojection screen.

Third, due to the external illumination nature of reflectivemicrodisplays such as LCoS, DLP and transmissive microdisplays such asLCD, the beam angle for each pixel can be narrower than with emissivemicrodisplays such as OLED or microLED which can allow to provide anoptical system with less stray light and be more efficient whileproviding the same or higher brightness to the viewer.

Fourth, due to the external illumination nature of reflective andtransmissive microdisplays much higher brightness is achievable thanwith emissive microdisplays which have the physical pixels emit thelight themselves like OLEDs and microLEDs or with LCD displays whichmakes it challenging to have them provide enough brightness, especiallyas the field of view and magnification of the display increases, or foraugmented reality head-mounted displays where there can be a lot oflight loss in the optical system.

In FIG. 3, a single 16:9 aspect ratio microdisplay or display is splitinto two parts, for example 1920×1080 pixel microdisplay or displaysplit into a small 1080×1080 pixel high resolution part 301 and a large840×1080 pixel low resolution part 302 (the latter may then be opticallyflipped 90 degrees for a better aspect ratio).

Using optical or optical and mechanical and also optionally digitalmethods, the parts 301 and 302 can be resized and superimposed on eachother 305 and the large low resolution part 303 can be masked where thesmall high resolution part 304 is and where they overlap.

The masking can further be made more seamless by blending the edgesoptically or digitally by making the transition less abrupt with adigital resolution falloff in the high resolution small image or dimmingthe pixels with a falloff on both images.

The brightness levels between the two parts may be balanced opticallysuch as with neutral density filters or digitally.

Look to FIGS. 4A and 4B. To be able to use the same projector 401, 411for each part which have a different size and position on the finalprojected image 405, with the first method from FIG. 2, the beam of theprojector is steered with a steering element optomechanically oroptically, such as, but not limited to, a rotating mirror or beamsplitter 402, 412 and an optional mirror 413, to one of two opticalelements 403, 404, 414, 415 for each frame. In case of using a beamsplitter instead of a rotating mirror as the steering element, each beameach frame may be blocked or let to pass accordingly before, inside orafter the optical element 403, 404, 414, 415 with an optical ormechanical shutter such as an LCD shutter in order to prevent 403, 414and 404, 415 from receiving the same beam of every frame instead of thedifferent beams of different consecutive frames. This is of course notneeded if a polarizer beam splitter (e.g., reflective polarizer beamsplitter) is used and the polarization of the beam can be controlledeach frame before it reaches the beam splitter, such as with aswitchable liquid crystal polarization rotator.

To be able to use the same projector 401, 411 for each part which have adifferent size and position on the final image 405 on the screen 418,with the second method from FIG. 3, the beam of the projector 401, 411is steered with a steering element such as, but not limited to, a beamsplitter or a mirror on an image plane 402, 412 and an optional mirror413, to two optical elements 403, 404, 414, 415. In case of using a beamsplitter and not a mirror on an image plane, each beam is then maskedaccordingly before, inside or after the optical element 403, 404, 414,415 with an optical masking element such as a stencil. The mirror orstencil may be on an image plane to create a sharp cut.

Steering element 402, 412 may be, but is not limited to, a mirror,mirrors, beam splitter and optical or mechanical shutter or shutterscombined with one of the above.

The optical element 403, 404, 414, 415 may be, but is not limited to,one of the following, or a combination of: lenses, mirrors, prisms,free-form mirrors.

One of the optical elements 404, 415 may create a narrow beam 417 andthe other optical element 403, 414 a comparably wide beam 416.

Looking to FIG. 4B, the projector 411 creates a projection beam that issplit with a beam splitter (such as half silvered mirror or polarizerbeam splitter) 412 into two identical projection beams going indifferent directions. One is directed to optics 414 which create a widebeam, while the other goes through a mirror 413 to another optics 415which creates a narrow beam. The wide beam optics 414 create the lowerresolution image beam 416. The narrow beam optics 415 creates the higherresolution image beam 417. Both the lower 416 and higher 417 resolutionbeams are projected onto the viewer's retina or screen 418.

Masking of the area of the wide beam 416 where the narrow beam 417 iscan be achieved, again, digitally by having black pixels displayedthere, or optically, for example by having a stencil on an image planesomewhere inside, before or after the optics to physically (optically)mask off that part of the projection beam.

Then, optionally, the positioning of the small image of the narrow beamcan be achieved with, but not limited to one or more of the following:actuators with mirrors, galvanometer scanners, actuators with wedge(Risley) prisms, actuators with tilting or shifting lenses, as seen inFIG. 5A.

The two beams are projected onto the same screen as seen in FIG. 4B orfirst optically combined, such as with a beam splitter, projected onto ascreen and viewed directly, or through, but not limited to, an eyepieceor waveguide. This is seen in the beam steering elements 519 and 518 ofFIG. 5B.

Looking to FIG. 5F, the optical elements are shown. The projector 5111sends the projection beam to a beam splitter or a rotating mirror 5112that sends projection beams to the wide beam optics 5114 and to a mirror5113 that redirects the projection beam to the narrow beam optics 5115.From the narrow beam optics 5115, the beam is sent to a mirror 5119 andthen to a beam combiner 5118 (e.g., beam splitter) to combine with theoutput of the wide beam optics 5114. From the beam combiner 5118, thewide beam 5116 and the narrow beam 5117, are sent as a combinedprojection beam to the viewer's retina or the screen 5110. In case ofusing a beam splitter instead of a rotating mirror as the steeringelement, each beam each frame may be blocked or let to pass accordinglybefore, inside or after the optical element 5114, 5115 with an opticalor mechanical shutter such as an LCD shutter in order to prevent 5114and 5115 from receiving the same beam of every frame instead of thedifferent beams of different consecutive frames. This is of course notneeded if a polarizer beam splitter is used and the polarization of theimage can be controlled each frame before it reaches the beam splitter,such as with a switchable liquid crystal polarization rotator.

One difference between FIGS. 5F and 5G is that projection beams areillustrated as two lines rather than one before reaching the opticalmasking elements. This is done to illustrate how the projection beam ismasked/cropped by the optical masking elements 5130, 5131.

Looking to FIG. 5G, the optical elements are shown, for processing theimage structure in FIG. 3. The projector 5121 sends the projection beamto a beam splitter 5122 that sends two identical projection beams, oneto a mirror 5123 first, to a stencil (a physical barrier to hide part ofthe image) 5130, 5131. The stencils may be on an image plane to create asharp cut, so can also be inside the optics (5124 and 5125), or afterthe optics.

The beams leave from the stencils 5130, 5131 to the wide beam optics5124 and to the narrow beam optics 5125. From the narrow beam optics5125, the beam is sent to a mirror 5129 and then to a beam combiner 5128(e.g., beam splitter) to combine with the output of the wide beam optics5124. From the beam combiner 5128, the wide beam 5126 and the narrowbeam 5127, are sent as a combined projection beam to the viewer's retinaor screen 5120.

Looking to FIG. 5H we see a head-mounted display embodiment which uses asingle projector 5151 for both screens (for both eyes). First theresolution of the microdisplay or display is split between eyes, theneach frame is used for one projection (large or small image). Forexample with a 240 Hz DLP projector this provides 120 Hz refresh rateper image per screen.

The projector 5151 sends the beam to a beam splitter 5160 that sends twoidentical beams, one reflected from a mirror 5182 first, to the stencils5161, 5171 that mask off the portion of the image not destined for thespecific eye. In one embodiment, the stencils 5161, 5171 could beshutters such as an LCD shutter or LCD pi-cell, so each frame will besent to one optics and blocked for the rest of the optics 5154, 5155,5174, 5175, such as in the instance seen in FIG. 2. In anotherembodiment, the stencils 5161, 5171 could be removed so each frame thewhole image will be sent to one optics and blocked for the rest of theoptics 5154, 5155, 5174, 5175, such as in the instance seen in FIG. 2.For example with a 240 Hz DLP projector this provides 60 Hz refresh rateper image per eye.

The left stencil (top in the diagram) 5161 sends the beam to a secondbeam splitter 5152 which send two identical beams, one to a mirror 5153first, to the two LCD shutters 5162, 5163 for the FIG. 2 embodiment. TheLCD shutters 5162, 5163 could be replaced with stencils (a physicalbarrier to hide part of the projection beam) for the FIG. 3 embodiment.The stencils may be on an image plane to create a sharp cut, so can alsobe inside the optics (5154 and 5155), or after the optics.

The beams leave from the shutters (or stencils) 5162, 5163 to the widebeam optics 5154 and narrow beam optics 5155. From the narrow beamoptics 5155, the beam is sent to a mirror 5159 and then to a beamcombiner 5158 (e.g., beam splitter) to combine with the output of thewide beam optics 5154. From the beam combiner 5158, the wide beam 5156and the narrow beam 5157, are sent as a combined beam to the screen 5150or viewer's retina.

The right stencil (bottom in the diagram) 5171 sends the beam to asecond beam splitter 5172 which sends two identical beams, one to amirror 5173 first, to the two LCD shutters 5180, 5181 for the FIG. 2embodiment. The LCD shutters 5180, 5181 could be replaced with stencils(a physical barrier to hide part of the projection beam) for the FIG. 3embodiment. This has to be in an image plane to create a sharp cut, socan also be inside the optics (5174 and 5175), or after the optics.

The beams leave from the LCD shutters (or stencils) 5180, 5181 to thewide beam optics 5174 and narrow beam optics 5175. From the narrow beamoptics 5175, the beam is sent to a mirror 5179 and then to a beamcombiner 5178 (e.g., beam splitter) to combine with the output of thewide beam optics 5174. From the beam combiner 5178, the wide beam 5176and the narrow beam 5177, are sent as a combined beam to the screen 5170or viewer's retina.

Due to persistence of vision with the method in FIG. 2 and masking withthe method in FIG. 3 the two parts appear as one uniform projected image604 in FIG. 6B.

In FIG. 6A, the illustration shows rectangles representing individualpixels 601. FIG. 6B shows an illustration with individual pixelsdisplaying an actual image 604.

Since the small high resolution part 603, 606 in the final projectedimage 601, 604 can be smaller than it could be without the use of thesemethods, the variable-resolution screen method and apparatus describedhere allows to achieve more resolution visible in one or more parts ofthe projected image than is possible with the projector when usedwithout the methods described here.

This allows to achieve a variable-resolution screen, such as ahead-mounted display screen which uses one projector or one per eye witha high pixel or scanline density in the center of the field of view ofthe viewer and less in the periphery.

Optionally, by adding eye tracking via, but not limited to, gazetracking cameras or electrodes, the small high resolution part 603, 606can be positioned on the final projected image 601, 604 on the large lowresolution part 602, 605 where the viewer's foveal view is at any givenpoint in time. This allows to always have more pixels or scanlinesconcentrated in the foveal and optionally also in the near peripheralview of the viewer at any given point in time.

Optionally the positioning of the large low resolution part 602, 605 canbe achieved the same way the positioning of the small high resolutionpart 603, 606, for example to have pixels only in the field of view ofthe viewer's eye and not the total field of view of the viewer whichtakes into account eye rotations.

There can also be more than two parts, such as three, one for the fovealview, one for near peripheral and one for far peripheral and they can becombined and optionally positioned the same way as mentioned above.

Those skilled in the art will understand that the order of some elementscan be changed and more can be added, such as steering both large andsmall images together after they are optically combined, or adding moreelements for creating more small or large parts on the final projectedimage.

Shifting with Optical Slabs or Mirrors a High Resolution Small Image orNarrow Projection Beam Over a Low Resolution Large Image or WideProjection Beam

In another embodiment, a variable-resolution screen is achieved byshifting/offsetting a small and high resolution image or projection beamover a large and low resolution image or projection beam with opticalslabs or mirrors.

To achieve a variable-resolution screen a single display technology suchas a microdisplay or display or a single video projector such as asingle illuminated microdisplay, display, LBS (laser beam steering)projector or other type of video projector (from now on referred to as“projector”) 401, 411, 501, 511, 521, 551, 2301, 5111, 5121, 5151 isoperated at fast refresh rates. In FIG. 2, each consecutive frame (framen+1) the microdisplay, display or projector is used to either display orproject a small high resolution part 204 or parts of the final image 205or a large low resolution part 203 or parts of the final image 205 bysharing the refresh rate of the frames 201, 202 and final image 205between the latter's two or more parts 203, 204. Persistence of visionblends the two parts 203, 204 into one final image 205.

FIG. 3 shows an alternative embodiment, to achieve a variable-resolutionscreen a single display technology such as a microdisplay, display or asingle video projector such as a single illuminated microdisplay,display, LBS (Laser beam steering) projector or other type of videoprojector (from now on referred to as “projector”) is optically splitinto two or more parts 301, 302. This method allows a small highresolution part or parts 304 to use more pixels on the final image 305by sacrificing the resolution of a large low resolution part 303 orparts.

The two methods can also be combined to allow to create more parts onthe final image or to allow to create two or more final images bysharing both the resolution and refresh rate of the microdisplay,display or projector between the parts, such as for using a singlemicrodisplay, display or projector to create final images for both eyesin a head-mounted display.

In FIG. 3, a single 16:9 aspect ratio microdisplay or display is splitinto two parts, for example 1920×1080 pixel microdisplay or displaysplit into a small 1080×1080 pixel high resolution part 301 and a large840×1080 pixel low resolution part 302 (the latter may then be opticallyflipped 90 degrees for a better aspect ratio).

Using optical or optical and mechanical and also optionally digitalmethods, the parts 301 and 302 can be resized and superimposed on eachother and the large low resolution part 303 can be masked where thesmall high resolution part 304 is and where they overlap.

The masking can further be made more seamless by blending the edgesoptically or digitally by making the transition less abrupt with adigital resolution falloff in the high resolution small image 304 ornarrow beam or dimming the pixels with a falloff on both images orbeams.

The brightness levels between the two parts may be balanced opticallysuch as with neutral density filters or digitally.

Look to FIGS. 4A and 4B. To be able to use the same microdisplay,display or projector 401, 411 for each part which have a different sizeand position on the final image 405, with the first method from FIG. 2,the image of the microdisplay or display or the beam of the projector issteered with a steering element optomechanically or optically, such as,but not limited to, a rotating mirror or beam splitter 402, 412 and anoptional mirror 413, to one of two optical elements 403, 404, 414, 415for each frame. In case of using a beam splitter instead of a rotatingmirror as the steering element, each image or beam each frame may beblocked or let to pass accordingly before, inside or after the opticalelement 403, 404, 414, 415 with an optical or mechanical shutter such asan LCD shutter in order to prevent 403, 414 and 404, 415 from receivingthe same image or beam of every frame instead of the different images orbeams of different consecutive frames. This is of course not needed if apolarizer beam splitter is used and the polarization of the beam can becontrolled each frame before it reaches the beam splitter, such as witha switchable liquid crystal polarization rotator.

To be able to use the same microdisplay, display or projector 401, 411for each part which have a different size and position on the finalimage 405 with the second method from FIG. 3, the image of themicrodisplay or display or the beam of the projector 401, 411 is steeredwith a steering element such as, but not limited to, a beam splitter ora mirror on an image plane 402, 412 and an optional mirror 413, to twooptical elements 403, 404, 414, 415. In case of using a beam splitterand not a mirror on an image plane, each image or beam is then maskedaccordingly before, inside or after the optical element 403, 404, 414,415 with an optical masking element such as a stencil. The mirror orstencil may be on an image plane to create a sharp cut.

Steering element 402, 412 may be, but is not limited to, a mirror,mirrors, beam splitter and optical or mechanical shutter or shutterscombined with one of the above.

The optical element 403, 404, 414, 415 may be, but is not limited to,one of the following, or a combination of: lenses, mirrors, prisms,free-form mirrors.

One of the optical elements 404, 415 may create a small image or narrowbeam 417 and the other optical element 403, 414 a comparably large imageor wide beam 416.

In the embodiment in FIGS. 5A and 5B, the positioning of the small imageor narrow beam can be achieved with, but not limited to one or more ofthe following: optical slabs or mirrors 506, 519, 529, 559, 579, 2310,2311, 5119, 5129, 5159, 5179.

The two images or beams are optically combined, such as with a beamsplitter and viewed directly, or through, but not limited to, aneyepiece or waveguide.

Due to persistence of vision with the method in FIG. 2 and masking withthe method in FIG. 3 the two parts appear as one uniform image 604 inFIG. 6B.

Looking to FIG. 5E, we see a variant of FIG. 5B or 5F with two tiltingoptical slabs 2310 and 2311. The microdisplay, display or projector 2301creates the image or beam and sends the image or beam through a beamsplitter or a rotating mirror 2302. Two identical images or beams aresent from the beam splitter 2302. One image or beam is sent through thelow resolution, large image or wide beam optics 2304, where the highresolution portion is masked off in case of using the method in FIG. 3,and then to the beam splitter 2308, used here as a beam combiner. Theother image or beam is sent from the beam splitter 2302 to a mirror 2303to the high resolution, small image or narrow beam optics 2305, wherethe low resolution image is masked off in case of using the method inFIG. 3. From the small image or narrow beam optics 2305, the image orbeam is reflected off a mirror 2309 to two beam steering elements 2310,2311, to offset the small image or narrow beam in the axis (after itwill be combined with the beam combiner 2308) of the large image or widebeam. In this illustration the beam steering elements are two thickoptical slabs 2310, 2311 that rotate in X and Y axis respectively tooffset the image or beam in these two respective axis. The optical slabs2310, 2311 may each be substituted with a single mirror that rotates inboth axis or two rotating/tilting mirrors, to name a few possiblealternative embodiments. From the second optical slab 2311, the shiftedimage or beam travels to the beam splitter 2308, used here as a beamcombiner. From the beam splitter 2308, the low resolution, large imageor wide beam 2306 and the high resolution, small image or narrow beam2307 travel to the screen 2300 or viewer's retina.

In case of using a beam splitter instead of a rotating mirror as thesteering element 2302 and using the method in FIG. 2, each image eachframe may be blocked or let to pass accordingly before, inside or afterthe optical element 2304, 2305 with an optical or mechanical shuttersuch as an LCD shutter in order to prevent 2304 and 2305 from receivingthe same image of every frame instead of the different images ofdifferent consecutive frames. This is of course not needed if apolarizer beam splitter is used and the polarization of the image can becontrolled each frame before it reaches the beam splitter, such as witha switchable liquid crystal polarization rotator.

In FIG. 6A, the illustration shows rectangles representing individualpixels 601. FIG. 6B shows an illustration with individual pixelsdisplaying an actual image 604.

With tilting/rotating mirrors and rotating wedge (Risley) prisms, theprojection beam or image is steered and gets a perspective distortion,as seen in FIG. 7B, and some optical aberrations which get progressivelyworse as the image or beam is steered farther away from the center. Tofix the perspective distortion the image 702 may be pre-distorteddigitally which reduces the possible size of the high resolution smallimage and the number of utilized pixels significantly.

Also, if there is any inaccuracy or precision issues during positioning,it is visible as a very apparent distortion and seam as the digitaldistortion and image or projection beam do not match the currentpositioning by the mirror, prism or other tilting element, as seen inFIG. 8A.

FIG. 7A is an original image 701, and FIG. 7B is a perspective distortedimage 702.

In FIG. 8A, the correct image 801 is seen. In the image on the right 802(FIG. 8B) the digital imaging and image positioning and distortionmismatch which causes distortion and seam between the two image parts isseen.

With shifting/offsetting the image or beam instead, these issues do nothappen.

The beam or image can be shifted by, but not limited to, twotilting/rotating optical slabs, one for each axis, two dual axistilting/rotating mirrors such as Optotune™ MR-15-30-PS-25×25D or fourtilting/rotating mirrors (two per axis).

In FIGS. 9A-D, an optical slab 902 is a glass slab or a plastic polymerslab clear in the visible spectrum which allows to shift/offset an imageor projection beam 903.

Both an image as well as a projection beam 903 may be shifted with thismethod. The latter allows to have the slabs 902 relatively small whichcan direct the projection beam to projection optics which can produce alarge projected image not requiring much more magnification by theeyepiece lens, waveguide or similar optics in a head-mounted displaydevice.

However, an image may be shifted by this method as well when themagnification can be performed by the eyepiece optics, limited amount ofshifting is needed or limited amount of magnification is needed by theeyepiece lens, waveguide or similar optics.

In FIG. 9B we see a 20×20×20 mm PMMA (Poly(methyl methacrylate)) polymeroptical slab 902 with a collimated 5 mm wide 638 nm wavelength beam 903passing through it and being shifted. In this example the slab can tilt+34 degrees (the range is ±34 degrees) and offset the beam by up to 8.04mm. Considering a situation where such a beam later goes through aprojection lens and the 5 mm beam is meant to cover 20 degrees of thefield of view when looking through the eyepiece or waveguide, a 16.08 mmshift would allow to move the high resolution image which the beamcontains by over 64 degrees or more which is more than the average humancan comfortably rotate their eyes.

In FIG. 9B the optical slab 904 is tilted −34 degrees to offset the beam903 8.04 mm downwards.

Two of such slabs 902, 904 will be needed, as seen in FIG. 5E, rotatingin different axis to allow to shift the beam 903 in both axis, or havingan optical component such as a Dove prism or an identical mirrorassembly between two slabs 902, 904 allowing them to rotate in the sameaxis.

The illustration is just for example purposes and different materialsand sizes for the slabs 902, 904, dimensions for the beams 903 androtation ranges are possible.

Slight dispersion of an RGB image or projection beam 903 caused by theoptical slab 902, 904 can be compensated for by digitally offsettingeach color channel by several pixels accordingly. Since offsetting mayonly be required to be done only to one or two color channels withhigher refractive index, one or two color channels won't be able toreach the same offset on the edges of the image or projection beam 903which may be resolved by digitally or optically cropping the image orprojection beam 903 slightly on the edges so the pixels in each colorchannel can be offset as much as is required to undo the separation ofthe color channels caused by dispersion. This loss of pixels on theedges is still negligible compared to loss of pixels/detail due tocorrection of a perspective distortion from previous embodiments.

With the above example at the extreme ±34 degree slab tilt the angle ofrefraction at 445 nm wavelength is ±21.9 degrees and at 638 nmwavelength is ±22.1 degrees. This results in 0.06 mm dispersion betweenthe red and blue color channel of the image or projection beam 903.Assuming the resolution of this 5 mm wide image or projection beam 903is 1080 pixels by 1080 pixels, this amounts to 0.06×1080/5=12.96 pixels.Sacrificing 13 pixels on each edge of the beam 903 will allow to offsetthe color channels digitally to undo the effect of dispersion at anyangle.

Specifically looking to FIG. 9A, we see the beam 903 moving through theair 901 to the slab 902. Since the slab 902 is perpendicular to the beam903, the beam 903 goes straight through the slab 902.

In FIG. 9B, the slab 904 is tilted −34 degrees, causing the beam 903 tobe offset 8.04 mm downwards.

In FIG. 9C, the slab 902 is tilted +34 degrees, causing the beam 903 tobe offset 8.04 mm upwards.

In FIG. 9D, there are two views of a single slab 902, 904 superimposedover each other to illustrate how much the beam offsets from one angleto the other. The slab view 904 is tilted −34 degrees, causing the beam903 to be offset 8.04 mm downwards while slab view 902 is tilted +34causing the beam 903 to be offset 8.04 mm upwards. Thus the beam 903 maybe offset up and down, creating images or beams at most 16.08 mm apart.

As seen in FIG. 10A and FIG. 10B, the slabs 902, 904 can also be swappedwith 2d mirrors (dual axis tilting/rotating mirrors such as Optotune™MR-15-30-PS-25×25D) or two mirrors 1001, 1002 or 1003, 1004. This is asavings in cost traded off with bigger space requirements. On the otherhand, dispersion is not an issue with mirrors.

In FIG. 10A the mirrors are tilted at 45 degrees 1001, 1002 and 40degrees 1003, 1004 in FIG. 10B.

Two 2D mirrors rotating in two axis or four mirrors 1101, 1102, 1103,1104 may be used to shift the beam or image in two axis as seen in FIG.11.

In FIG. 11, both mirrors 1101, 1102 have a top-down view purely forillustrative purposes, the second set of mirrors 1103, 1104 are inanother axis.

FIG. 12 shows another embodiment. Either the second set of mirrors 1204,1205 can be flipped and rotated in another axis or to save space in oneaxis a Dove prism 1203 or an equivalent mirror assembly may be placedbetween the two 2d mirrors or mirror pairs 1201, 1202 and 1204, 1205 toflip the axis of the offset performed by the previous set and have themirrors and the components which shift/offset them in the same axis.

In FIG. 12 we see the path the ray travels when using the Dove prism1203 (The Dove prism proportions and angle are not accurate in thisdrawing, nor is the path the ray travels inside the Dove prism itself).

Since the smaller high resolution part in the final image can be smallerthan it could be without the use of these methods, thevariable-resolution screen method and apparatus described here allows toachieve more resolution visible in one or more parts of the final imagethan is possible with the display, microdisplay or projector when usedwithout the method described here.

This allows to achieve a variable-resolution screen, such as ahead-mounted display screen which uses as little as one microdisplay,display or projector or one per eye with a high pixel or scanlinedensity in the center of the field of view of the viewer and less in theperiphery.

By adding eye tracking via, but not limited to, gaze tracking cameras orelectrodes, the smaller high resolution part can be moved on the finalimage or screen on the bigger low resolution part where the viewer'sfoveal view is at any given point in time. This allows to always havemore pixels or scanlines concentrated in the foveal and optionally alsoin the near peripheral view of the viewer at any given point in time.

Optionally the positioning of the bigger low resolution part can beachieved the same way the positioning of the smaller high resolutionpart, for example to have pixels only in the field of view of theviewer's eye and not the total field of view of the viewer which takesinto account eye rotations.

There can also be more than two parts, such as three, one for the fovealview, one for near peripheral and one for far peripheral and they can becombined the same way as mentioned above.

Those skilled in the art will understand that the order of some elementsin the diagrams can be changed and more can be added, such as shiftingboth large and small images or beams together after they are opticallycombined, or adding more elements for creating more smaller or biggerparts on the final image.

Variable-Resolution Screen with No Moving Parts

In a further embodiment, a variable-resolution screen is achieved bycreating and digitally and optically positioning a small and highresolution image or projection over a large low resolution image orprojection with no mechanically moving parts.

The image source for the at least one large low resolution part 201 andat least one small high resolution part 202 can be the samemicrodisplay, display or projector with consecutive frames (frame n andframe n+1) distributed between the two or more parts 203, 204 of thefinal image or beam (see FIG. 2). Or the parts of the images of themicrodisplay, display or projector could be optically split into two301, 302 or more and allocated between the at least one large lowresolution part 303 and at least one small high resolution part 304, asin FIG. 3. Alternatively, the at least one large low resolution part andat least one small high resolution part can have a differentmicrodisplay, display or projector as image source each as in FIG. 13.

See FIG. 3, where a single 16:9 aspect ratio microdisplay or display issplit into two parts, for example 1920×1080 pixel microdisplay ordisplay split into a small 1080×1080 pixel high resolution part 301 anda large 840×1080 pixel low resolution part 302 (the latter may then beoptically flipped 90 degrees for a better aspect ratio).

The lack of mechanically moving parts provides several advantages:

First, eliminating moving parts eliminates the sensitivity to vibration,misalignment, mechanical failure, audible noise or any other issuesassociated with using mechanically moving parts.

Second, repositioning of the small high resolution part can take as lowas a few microseconds to a few milliseconds, based on the speed of theoptical masking element used as described below. By contrast it isdifficult to get actuators to rotate a mirror, prism or slab as fast asthe saccadic movement of the human eye while keeping such a motor assmall as possible for a wearable device.

Third, positioning takes equal amounts of time irrespective of the newposition the small high resolution part has to be positioned to.

At first, an image or projection beam is optically duplicated across thewhole or most of the screen or the viewer's retina or part of the screenthe human eye can rotate and focus at.

This can be achieved by, for example, the use of lens arrays. Forillustrative and purposes of showing an example a single or double sidedlens array is used, however a multi-element lens and/or lens array setupmay be used to reduce optical aberrations in the duplicated images orvideo projections.

FIG. 13 shows a two microdisplays, displays or projectors 1301, 1302embodiment using a lens array. The large image or wide beam is createdby the first microdisplay, display or projector 1301 and sent directly(or through a large image or wide beam optics) to the final image 1305.The second microdisplay, display or projector 1302 creates the smallimage or narrow beam, sending it to the lens array (or other duplicationelement) 1303 and then to an optical masking element 1304 to mask off(hide) the duplicates in the area outside of the one duplicate image tobe shown. The image or beam then proceeds to the final image 1305 whereit is combined with the large image or wide beam from the firstmicrodisplay, display or projector 1301.

FIG. 14 shows a similar embodiment, using a single microdisplay, displayor projector 1401. The image or beam proceeds from the microdisplay,display or projector 1401 to an image or beam steering element 1402. Thesteering element 1402 splits the image or beam, with the large imageportion of the final image or beam sent directly (or through a largeimage or wide beam optics) to the final image 1405 (in some embodiments,such as in FIG. 3, the image is masked to extract small and large imageportions accordingly first). The small image portion of the final imageor beam is sent to the lens array (or other duplication element) 1403,and then to the optical masking element 1404 to mask off (hide) theduplicates in the area outside of the one duplicate image to be shown.This small image or narrow beam is then combined with the large image orwide beam from the steering element 1402 to form the final image 1405.

FIG. 15 shows the simplest setup of how display, microdisplay orprojection beam can be duplicated this way and FIGS. 16A and 16B showthe simulated result.

In FIG. 15, the image source (display, microdisplay or projector) 1501sends the image to a lens 1502 which sends it to the aperture stop 1504.The image or beam then proceeds to the lens array 1503 and then to thescreen 1505 or viewer's retina.

FIGS. 16A and 16B show the simulated result. FIG. 16A is the originalimage from the display, microdisplay or projector 1601 and FIG. 16B isthe resulting image on the screen or viewer's retina 1602.

FIG. 17A shows a simple setup with one possible position of the opticalmasking element. 1701 is a microdisplay, display or projector, 1702 isthe light cone (beam) of a single pixel from 1701. 1703 is a simplifiedillustration of a multi-element lens. 1704 is the first lens array whichfocuses the pixel light cones (beams) to pixels on a LCD microdisplayoptical masking element 1705 on an intermediate image plane and 1706 isthe second lens array which again focuses the pixel light cones on thefinal image plane on a projection screen 1707 or the viewer's retina.The second lens array 1706 can also be replaced with other optics suchas an ordinary projection lens or eyepiece lens.

FIG. 17B shows a reflective microdisplay such as LCoS or DLP used forthe optical masking element. 1711 is a microdisplay, display orprojector generating the image or beam, 1712 is a light cone (beam) of asingle pixel from 1711, 1713 is a simplified illustration of amulti-element lens, 1714 is the first lens array which focuses the pixellight cones (beams) to pixels on a LCoS microdisplay optical maskingelement 1715 on an intermediate image plane and 1717 is the second lensarray which again focuses the pixel light cones (beams) on the finalimage plane on a projection screen 1718 or the viewer's retina. Apolarizer beam splitter or a PBS (polarizer beam splitter) cube 1716 isused to redirect the image or beam reflected off the LCoS microdisplayoptical masking element 1715 90 degrees to the side rather than back tothe first lens array. The second lens array 1717 can also be replacedwith other optics such as an ordinary projection lens or eyepiece lens.With DLP microdisplay a TIR or RTIR (total internal reflection) prismcan be used in place of the polarizer beam splitter or PBS (polarizerbeam splitter) cube 1716.

FIG. 17C shows that it is also possible to use a display, notmicrodisplay, such as an LCD display with its reflective and backlightlayers removed as the optical masking element. 1721 is a microdisplay,display or projector generating the image or beam, 1722 is a light cone(beam) of a single pixel from 1721, 1723 is a simplified illustration ofa multi-element lens, 1724 is the lens array which focuses the pixellight cones (beams) to pixels on a screen 1727 on an image plane behinda LCD display optical masking element 1726 by reflecting the beam with abeam splitter 1725. The image from second display 1728 also reflectsfrom the beam splitter thus both the second display and the screen areseen by the eye 1720 directly or through an eyepiece or waveguide 1729.Here the screen 1727 is used to display the small high resolution imageand the display 1728 is used to display the large low resolution imageof the final combined image combined by the beam splitter 1725.

Alternatively, it is also possible to use a LCD display with itsreflective and backlight layers removed as an optical masking elementwith a single microdisplay, display or projector (or two, as seen inFIG. 13) generating the image or beam without a second display 1728, asillustrated in FIG. 17D. In case of time-multiplexed approach asdescribed in FIG. 2 a beam splitter 1725 is also not necessary asillustrated in FIG. 20 and FIG. 21.

In one embodiment, the beam splitter 1725 is a reflective polarizer beamsplitter. In one embodiment, a first quarter wave plate (not shown) maybe positioned between beam splitter 1725 and screen 1728 and/or a secondquarter wave plate (not shown) may be positioned between beam splitter1725 and screen 1727. The quarter wave plates may rotate thepolarization of light reflecting off of the screen 1728 and screen 1727,respectively.

In the FIG. 17D illustration, the elements for generating a duplicatedimage or beam are not illustrated and are in 1731 which represents amicrodisplay, display or projector or two microdisplays, displays orprojectors with the wide beam and duplicated beam already opticallycombined as described in FIG. 13 and FIG. 14. Light cone (beam) of asingle pixel of the wide beam 1732 and light cone (beam) of single pixelof a duplicate beam 1733 both focus to pixels on a screen 1736 on animage plane behind a LCD display optical masking element 1735 by beingreflected from a beam splitter 1734. The screen 1736 is seen by the eye1738 directly or through an eyepiece or waveguide 1737.

In one embodiment, the beam splitter 1734 is a reflective polarizer beamsplitter. A first quarter wave plate (not shown) may be positionedbetween beam splitter 1734 and screen 1736. The quarter wave plate mayrotate the polarization of light reflecting off of the screen 1736.

In case of splitting a microdisplay, display or projector into two ormore parts as illustrated in FIG. 3 a beam splitter may be used and alsoa second screen may be used as illustrated in FIG. 17E.

In the next illustration, FIG. 17E, the elements for generating aduplicated image or beam are not illustrated and are in optical system1741 which represents one or more of a microdisplay, display orprojector or two microdisplays, displays or projectors, an imagesteering element, small and/or large image optical elements, an imageseparation element, and/or a beam combiner. In one embodiment, theoptical system 1741 includes one or more image source, an image steeringelement, a small image optical element, a large image optical element,and a beam combiner. The image steering element may be configured toseparate an image into a first image component and a second imagecomponent as well as to direct the first image component to a firstdestination and to direct the second image component to a seconddestination. The wide beam (e.g., representing a low resolution, largeimage) and duplicated beam (e.g., representing a plurality of duplicatesof a high resolution, small image) may already be optically combined asdescribed in FIG. 13 and FIG. 14 at the output of the optical system1741. The wide beam may be or represent a low resolution, large image,and the duplicated beam may be or represent a plurality of duplicates ofa high resolution, small image. The low resolution, large image and theplurality of duplicates of the high resolution, small image may havebeen optically combined into a combined image as described in FIG. 13and FIG. 14.

Light cone (beam) of a single pixel of the wide beam 1742 and light cone(beam) of a single pixel of a duplicate beam 1743 may focus to pixels ona second screen 1747 and screen 1746 respectively, the latter on animage plane behind an optical masking element 1745 (e.g., which may bean LCD display optical masking element or another type of opticalmasking element) by being reflected from a beam splitter 1744. The widebeam 1742 passes through the beam splitter 1744 and the duplicate beamgets reflected from the beam splitter 1744 instead due to these beamshaving different polarization (or in the case the beam splitter is aband pass filter or dichroic filter, having different lightwavelengths). In an example, the low resolution, large image may passthrough the beam splitter 1744 onto screen 1747, and the plurality ofduplicates of the high resolution, small image may be reflected off ofthe beam splitter 1744 and onto screen 1746. The optical masking element1745 may be positioned between the screen 1746 and the beam splitter1744, and may mask off one or more of the plurality of duplicates of thehigh resolution, small image such that a single duplicate of the highresolution, small image remains, as described above. The screens 1746and 1747 are optically combined by the beam splitter 1744 and seen bythe eye 1749 directly or through the eyepiece or waveguide 1748.Accordingly, beam splitter 1744 may recombine the single duplicate ofthe high resolution, small image with the low resolution, large image toproduce a variable resolution image that may be directed to the eyepieceor waveguide 1748 or focused directly onto the eye 1749 of a viewer.

In one embodiment, the beam splitter 1744 is a reflective polarizer beamsplitter. In one embodiment, a first quarter wave plate (not shown) maybe positioned between beam splitter 1744 and screen 1747 and/or a secondquarter wave plate (not shown) may be positioned between beam splitter1744 and screen 1746. The quarter wave plates may rotate thepolarization of light reflecting off of the screen 1747 and screen 1746,respectively, to permit the reflected light from the screen 1747 toreflect off of the beam splitter 1744 (reflective polarizer beamsplitter) and to permit the reflected light from the screen 1746 to passthrough the beam splitter 1744 (reflective polarizer beam splitter) andarrive at eye 1749 and/or eyepiece or waveguide 1748.

Alternatively, in case of splitting a microdisplay, display or projectorinto two or more parts as illustrated in FIG. 3 both a beam splitter andalso a second screen are not needed similarly to the case oftime-multiplexing as illustrated in FIG. 2, as illustrated in FIGS. 17Fand 17G.

In FIG. 17F the elements for generating a duplicated image or beam arenot illustrated and are in 1751 which represents a microdisplay, displayor projector or two microdisplays, displays or projectors with the widebeam and duplicated beam already optically combined as described in FIG.13 and FIG. 14.

Light cone (beam) of a single pixel of the wide beam 1752 and light cone(beam) of single pixel of a duplicate beam 1753 both focus to pixels ona screen 1756 on an image plane behind a LCD display optical maskingelement 1755 by being reflected from a beam splitter 1754. The screen1756 is seen by the eye 1759 directly or through the eyepiece orwaveguide 1757.

To be able to pass both the wide and duplicated beams through the sameLCD display optical masking element but use the optical masking elementfor blocking the duplicated beams, instead of a traditional LCD displayoptical masking element a switchable liquid crystal polarization rotatordisplay is used which is an LCD display optical masking element withoutpolarizers. A single polarizer 1758, not two as on LCD display opticalmasking elements and displays, is placed before the viewer's eye 1759and in front of the eyepiece or waveguide 1757 or somewhere before it orleft on the LCD display optical masking element 1755.

The wide beam in this instance is not polarized or in the polarizationstate the polarizer 1758 is not going to filter out after the wide beampasses through the switchable liquid crystal polarization rotator/LCDdisplay optical masking element 1755. The duplicated beam gets masked asexpected by the LCD display optical masking element 1755 and thepolarizer 1758 while the wide beam does not or gets masked where theduplicated beam is not masked.

In one embodiment, the beam splitter 1754 is a reflective polarizer beamsplitter. In one embodiment, a first quarter wave plate (not shown) maybe positioned between beam splitter 1754 and screen 1756. The quarterwave plate may rotate the polarization of light reflecting off of thescreen 1756.

As mentioned previously the beam splitter 1754 is not necessary and usedfor reasons such as decreasing the physical dimensions of the apparatus.FIG. 17G illustrates the same system as FIG. 17F sans the beam splitter1754.

In FIG. 17G the elements for generating a duplicated image or beam arenot illustrated and are in 1761 which represents a microdisplay, displayor projector or two microdisplays, displays or projectors with the widebeam and duplicated beam already optically combined as described in FIG.13 and FIG. 14.

Light cone (beam) of a single pixel of the wide beam 1762 and light cone(beam) of single pixel of a duplicate beam 1763 both focus to pixels ona screen 1765 on an image plane behind a LCD display optical maskingelement 1764. The screen 1765 is seen by the eye 1768 directly orthrough the eyepiece or waveguide 1766.

To be able to pass the wide and duplicated beams through the same LCDdisplay optical masking element but use the optical masking element forblocking the duplicated beams, instead of a traditional LCD displayoptical masking element a switchable liquid crystal polarization rotatordisplay is used which is an LCD display optical masking element withoutpolarizers. A single polarizer 1767, not two as on LCD display opticalmasking element and displays, is placed before the viewer's eye 1768 andin front of the eyepiece or waveguide 1766 or somewhere before it orleft on the LCD display optical masking element 1764.

The wide beam in this instance is not polarized or in the polarizationstate the polarizer 1767 is not going to filter out after the wide beampasses through the switchable liquid crystal polarization rotator/LCDdisplay optical masking element without the polarizers 1764. Theduplicated beam gets masked as expected by the LCD display opticalmasking element without the polarizers 1764 and the polarizer 1767 whilethe wide beam does not or gets masked where the duplicated beam is notmasked.

With the optical masking element it is possible to show one of theduplicate images at a time, however with digital manipulation of thesource frame it is possible to have a digital and optical reconstructionof the original image visible anywhere on the duplicated image arrayarea while hiding everything else with a positional accuracy up to thepixel resolution of the optical masking element and positioning speedequal to the few microsecond to millisecond pixel switching speed of theoptical masking element.

As an example, let's consider each duplicated image being made up for 4parts, 1, 2, 3 and 4, as illustrated in FIG. 18, item 1801. In FIG. 18,the items on the left column illustrate these parts as squares withnumbers while the right column uses actual image parts.

In FIG. 18, 4 of such duplicate images are stacked 1802. If we wanted todisplay one duplicate in the middle of this array 1802, we wouldn't beable to as illustrated in item 1803.

However, if we take the original image 1801, partition it into 4 piecesdigitally and reposition those pieces digitally as in 1804, then we willget the result we want even though we are displaying parts of 4duplicates at once.

The duplicates are then masked and the original image 1801 properlyreconstructed by optical and digital methods as seen in 1805.

Since the optical masking elements discussed such as DLP, LCoS or LCDmicrodisplays or LCD displays are usually not double the resolution ofthe lens array but much more, the images can be partitioned into 4rectangles and rearranged digitally not only at the middle of the imagebut at any desired location on the image as seen in 1901, 1902, 1903,1904 in FIG. 19 with a possible limitation being the resolution of thesource image display, microdisplay or projector and the resolution ofthe optical masking element. Of course the visible portion from theoptical masking element cannot be larger than the size of a singleduplicate image from the array.

Head-Mounted Display Embodiments

In some embodiments, a head-mounted display includes an image sourceconfigured to output one or more image components and one or moreoptical element configured to receive the one or more image componentsand output one or more images onto a projection screen. In someembodiments, as described in greater detail herein above and below, theone or more image components comprise a high resolution, small imagecomponent and a low resolution, large image component and the one ormore optical element is configured to receive the high resolution, smallimage component and output a high resolution, small image and to receivethe low resolution, large image component and output a low resolution,large image. The high resolution, small image and the low resolution,large image may appear as a variable resolution image on the projectionscreen. In other embodiments, the one or more image components do notinclude a high resolution, small image component and a low resolution,large image component. For example, the image source may only output asingle image component or type of image component, or the image sourcemay output other combinations of image components than a highresolution, small image component and a low resolution, large imagecomponent. In such embodiments, the image that appears on the projectionscreen may not be a variable resolution image.

For embodiments in which the one or more image components include a highresolution, small image component and a low resolution, large imagecomponent, the one or more optical element may comprise a small imageoptical element configured to receive the high resolution, small imagecomponent and a large image optical element configured to receive thelow resolution, large image component. The small image optical elementand the large image optical element may share one or more of theirconstituents.

In some embodiments, the small image optical element includes aduplication element and an optical masking element, as described infurther detail herein above and below. The duplication element receivesthe high resolution, small image and outputs a plurality of duplicatesof the high resolution, small image. The optical masking element masksoff one or more of the plurality of duplicates of the high resolution,small image such that at least portions of one or more duplicates of thehigh resolution, small image remain, wherein at least the portions ofthe one or more duplicates of the high resolution, small image form(together or alone) a complete single duplicate of the high resolution,small image that is focused onto a target position on the projectionscreen.

In some embodiments, the duplication element comprises a beam splitterand a lens or a lens array. In some embodiments, the duplication elementcomprises a beam splitter array and a lens or a lens array. In someembodiments, the duplication element comprises at least one of amulti-element lens or a lens array configured to reduce opticalaberrations in the plurality of duplicates of the high resolution, smallimage.

In some embodiments, the head-mounted display further includes an imagesteering element configured to control a placement of the highresolution, small image on the projection screen. In some embodiments,the head-mounted display comprises a liquid crystal prism. In someembodiments, the liquid crystal prism is a polarization or diffractiongrating with a liquid crystal polarization rotator. In some embodiments,the image steering element comprises a birefringent element and liquidcrystal polarization rotator.

In some embodiments, at least one of the high resolution, small image orthe low resolution, large image is a light field image.

In some embodiments, pixel information displayed by the high resolution,small image and a region of the low resolution, large image thatcorresponds to the high resolution, small image may be used together toenhance a contrast or bit depth of the high resolution, small image asit appears in the variable resolution image on the projection screen.For example, the high resolution, small image may be superimposed overthe region of the low resolution, large image that matches the highresolution, small image, but at a lower resolution. The combination ofdata from the high resolution, small image and the low resolution, largeimage at the region of the superimposition can provide an increasedcontrast and/or an increased bit depth (e.g., increased color depth).

In some embodiments, the image source comprises a separate display ormicrodisplay panel for each color channel of a plurality of colorchannels. In such an embodiment, the head-mounted display may furtherinclude at least one of a trichroic prism, an X-cube prism or a dichroicfilter to optically combine the plurality of color channels.

In some embodiments, the projection screen is a viewer's retina. In someembodiments, the head-mounted display comprises the projection screen.The projection screen may be a rear projection screen in front of orpart of an emissive or transmissive display panel of the head-mounteddisplay. The projection screen may be configured to scatter light fromthe emissive or transmissive display panel to cause a visible image fromthe projection screen and the emissive or transmissive display panel tobe on a same focal plane. In some embodiments, the emissive ortransmissive display panel is a liquid crystal display (LCD) panel, anorganic light emitting diode (OLED) display panel, or a micro-LEDdisplay panel.

In some embodiments, the head-mounted display further includes anemissive or transmissive display panel configured to provide increasedcontrast or color depth for the one or more images. The head-mounteddisplay further includes the projection screen, wherein the projectionscreen is a rear projection screen in front of or part of the emissiveor transmissive display panel. The emissive or transmissive displaypanel may be a liquid crystal display (LCD) panel, an organic lightemitting diode (OLED) display panel, or a micro-LED display panel.

In some embodiments, the projection screen is a component of thehead-mounted display. In other embodiments, the projection screen is notpart of the head-mounted display, and is an external screen viewed by awearer of the head-mounted display through the head-mounted display. Insome embodiments, the projection screen associated with the head-mounteddisplay is a rear projection screen. In some embodiments, the projectionscreen is biaxially curved.

In some embodiments, the head-mounted display further includes aneyepiece configured to view the projection screen, wherein the eyepiececomprises a liquid crystal lens to provide variable image focus based oneye tracking data.

In some embodiments, the head-mounted display further includes a liquidcrystal display, a liquid crystal microdisplay, or a liquid crystalshutter array configured to phase modulate at least one image of the oneor more images.

In some embodiments, the head-mounted display is configured to increasea resolution of at least one image of the one or more images withintentional image distortion.

In some embodiments, the head-mounted display further includes a pixelshifting element configured to perform pixel shifting to increase aresolution of at least one image of the one or more images output by theimage source(s). The pixel shifting element may be a liquid prism, aliquid crystal prism, or a liquid crystal microprism array. The pixelshifting element may also be or include a liquid crystal polarizationrotator and a birefringent optical element before or after the liquidcrystal polarization rotator.

In some embodiments, the head-mounted display further includes anoptical masking element configured to display a copy of at least oneimage of the one or more images to provide at least one of an increasedcontrast or an increased color depth for the at least one image.

In some embodiments, a resolution of the one or more images is increasedbased on changing an aspect ratio of the one or more images.

In some embodiments, the projection screen is biaxially curved and bothsteers and scatters a projection beam comprising the one or more images.

In some embodiments, the one or more image components comprises aplurality of consecutive frames. The head-mounted display may use theplurality of consecutive frames to enhance a contrast or a bit depth ofthe one or more images.

In some embodiments, the head-mounted display is a head-mountedprojective display. In such embodiments, the image source may comprise aseparate display or microdisplay panel for each color channel of aplurality of color channels. The head-mounted display may furtherinclude at least one of a trichroic prism, an X-cube prism or a dichroicfilter to optically combine the plurality of color channels.

The above optical designs can work for many different types of image andvideo displays. In head-mounted displays, the small space requirementspresent additional challenges.

FIG. 20 shows a direct embodiment of the mounting of the presentinventions in a head-mounted display. The variable-resolution optics2003 as shown in FIGS. 4A, 4B, 5, 13, 14, 17 produces the highresolution small image 2005 and the low resolution large image 2006 thatare sent directly to the screen 2004. A human eye 2001 looks through alens 2002 or other optics that collects the light 2007 from the image onthe screen 2004.

FIG. 21 shows an indirect embodiment of the mounting of the presentinventions in a head-mounted display. The variable-resolution optics2103 as shown in FIGS. 4A, 4B, 5, 13, 14, 17 produces an image that isreflected off of a mirror 2108. The high resolution small image 2105 andthe low resolution large image 2106 from the mirror 2108 are sent to thescreen 2104. A human eye 2101 looks through a lens 2102 or other opticsthat collects the light 2107 from the image on the screen 2104.

FIG. 22 shows an indirect embodiment with a beam splitter of themounting of the present inventions in a head-mounted display. Thevariable-resolution optics 2203 as shown in FIGS. 4A, 4B, 5, 13, 14, 17produces an image that is reflected off of a mirror 2208. The mirror2208 reflects the light to a beam splitter 2209 which reflects the highresolution small image 2205 and the low resolution large image 2206 ontothe screen 2204. A human eye 2201 looks through a lens 2202 or otheroptics and through the beam splitter 2209 to see the light 2207 from theimage on the screen 2204.

In one embodiment, the beam splitter 2209 is a reflective polarizer beamsplitter. In one embodiment, a quarter wave plate (not shown) may bepositioned between beam splitter 2209 and screen 2204. The quarter waveplate may rotate the polarization of light reflecting off of the screen2204 to permit the reflected light from the screen 2204 to pass throughthe beam splitter 2209 (reflective polarizer beam splitter) and arriveat eye 2201 and/or eyepiece or waveguide 2202.

FIG. 23 illustrates an embodiment that uses a combination of a firstoptical source that outputs an image onto a screen to show a firstportion of a variable resolution image and second optical source to showa second portion of the variable resolution image. In one embodiment,the first optical source is a projector that projects the image onto aprojection screen, and the second optical source is a display ormicrodisplay.

An optical system 2341 may include the elements for generating aduplicated image or beam, as discussed above. The optical system mayinclude, for example, an image source such as a microdisplay, display orprojector, and may further include a lens array that produces aduplicated beam (e.g., duplicates of a high resolution, small image).

A light cone (beam) of a single pixel of a duplicate beam 2343 may focusto pixels on a screen 2346. The screen 2346 may be on an image planebehind an optical masking element 2345 (e.g., which may be an LCDdisplay optical masking element or another type of optical maskingelement) by being reflected from a beam splitter 2344. Accordingly, aplurality of duplicates of a high resolution, small image may bereflected off of the beam splitter 2344 onto screen 2346. The opticalmasking element 2345 may be positioned between the screen 2346 and thebeam splitter 2344, and may mask off one or more of the plurality ofduplicates of the high resolution, small image such that a singleduplicate of the high resolution, small image remains, as describedabove.

A second image source 2347 may be a display or microdisplay, such as anorganic light emitting diode (OLED) display, a liquid crystal display(LCD), or other screen display.

The second image source 2347 may output a low resolution, large image(represented by a single beam 2342), which may reflect off of the beamsplitter 2344 toward an eye 2349, eyepiece and/or waveguide 2348. Thesingle remaining duplicate of the high resolution, small image(represented by a single beam 2370) may pass through the beam splitter2344 toward the 2349, eyepiece and/or waveguide 2348. The singleremaining duplicate of the high resolution, small image may merge with(e.g., be superimposed onto) the low resolution, large image to form avariable resolution image that may be directed to the eyepiece orwaveguide 2348 or focused directly onto the eye 2349 of a viewer.

Reflective and transmissive microdisplays or displays such as DLP, LCoSand LCD have opaque or non-reflective gaps between each individual pixelor subpixel. If such a microdisplay or display is used as an opticalmasking element and is placed exactly on an intermediate image plane andthe pixel size of the image projected onto it is smaller or close to thesize of the pixel gap, resolution will be lost due to some pixels beingcompletely or partially projected on these opaque or non-reflectivegaps. This can cause a “screen-door effect,” in which a grid ofhorizontal and vertical black lines may appear between pixels.Furthermore, if resolution is added, any screen-door effect from theoptical masking element microdisplay or display will remain. Onepossible solution to this problem is having the optical masking elementslightly offset to the intermediate image plane. For example, theoptical masking element may be offset from the focal plane of the smallimage optical element, may be offset from a focal plane of a large imageoptical element, or may be offset from a focal plane of an opticalsystem described herein. This does have the side effect of de-focusingthe mask and mask edges. However, as has been mentioned previously, thedefocusing of the mask and/or mask edges may actually be a desiredeffect in some embodiments.

FIG. 24A illustrates a source image, in accordance with an embodiment.FIG. 24B illustrates the source image of FIG. 24A on an optical maskingelement that is on an intermediate image plane, in accordance with anembodiment. As shown, there are gaps between pixels that show up as agrid of horizontal and vertical black lines. FIG. 24C illustrates thesource image of FIG. 24A on an optical masking element that is slightlyoffset from the intermediate image plane (focal plane), in accordancewith an embodiment. As shown, the screen-door effect is minimized.

Instead of having the plurality of duplicates of a high resolution,small image on the optical masking element simply be masked to onlyreflect or transmit one duplicate, the duplicate or duplicates which areto be reflected and transmitted can have their corresponding pixels onthe optical masking element also be displaying the same image, albeit ata lower resolution. Similarly, the optical masking element (or adifferent optical masking element) may not simply fully transmit orreflect the low resolution, large image but may also display the samelow resolution, large image. This may serve two purposes:

-   -   1. By having the same image spatially modulated twice, albeit on        the optical masking element microdisplay or display at a        possibly lower resolution, the contrast of the final image can        be enhanced. If the optical masking element is slightly offset        to the intermediate image plane, the low resolution version of        the image displayed by the optical masking element will become        blurred but also benefit from not having sharp corners on its        pixels. The resulting effect is very similar to one achieved by        a technology in LCD TVs and monitors called “full-array local        dimming”. This may be used both for the high resolution, small        image as well as low resolution, large image.    -   2. Since the resolution of the optical masking element may be        significantly higher than the LED array inside full-array local        dimming LCD TVs and monitors, and also since the resolution of        the low resolution, large image and the resolution of the        optical masking element may be the same, and also since human        vision has lower acuity for chroma (color) than for luminance        (brightness), the optical masking element may also be used not        only for enhancing contrast but also enhancing color depth (bit        depth) of the final image. The resulting effect is very similar        to one achieved by a video encoding and decoding technique of        having lower resolution for chroma (color) than luminance        (brightness) called “chroma sub sampling.”        This may be used both for the high resolution small image as        well as low resolution large image.

FIG. 25 shows how spatially modulating an image twice or spatiallymodulating two images which store different bit depth (color depth)information of an original image, from left to right, produces a highercontrast and/or higher bit depth (color depth) image on the right. InFIG. 25, an original image 2505 is passed through an optical mask. Theoptical mask may be a display or microdisplay that displays a copy ofthe image 2510. The original image 2505 and the copy of the image 2510may be combined to form merged image 2515, which may have an improvedcontrast and/or an improved color depth (bit depth) as compared to theoriginal image 2505. FIG. 26 shows, from left to right, how spatiallymodulating a high resolution, small image again or spatially modulatinga different bit depth (color depth) information of the high resolution,small image on the optical masking element, where it is displayed at alower resolution, produces a higher contrast and/or higher bit depth(color depth) image on the right. As shown, a single duplicate of a highresolution, small image 2605 may be output by a lens array. An opticalmasking element may mask off other duplicates of the high resolution,small image, and may also display a copy of the high resolution, smallimage 2610. The copy of the high resolution, small image 2610 may have alower resolution than the duplicate of the high resolution, small image2605. The single duplicate of the high resolution, small image 2605 maybe combined with the copy of the high resolution, small image 2610 toform a merged high resolution, small image 2615, which may have animproved contrast and/or an improved color depth (bit depth) as comparedto the single duplicate of the high resolution, small image 2605 alone.

If the microdisplay, display or projector has subpixels then it maystore first set of bit depth (color depth) of the final image and theoptical masking element microdisplay or display may display theremainder. For example an OLED microdisplay may display 8 bit pixels andthen the optical masking element LCoS, LCD or DLP microdisplay may beused to modulate the pixels again to reach 10 bit or more bit depth(color depth) on the final image.

If the microdisplay, display or projector operates color-sequentially,then it may store first set of bit depth (color depth) of the finalimage and the optical masking element microdisplay or display may againdisplay the remainder. For example an LCoS microdisplay may display 8bit pixels and then the optical masking element LCoS, LCD or DLPmicrodisplay may be used to modulate the pixels again to reach 10 bit ormore bit depth (color depth) on the final image.

Furthermore, as has been mentioned previously, a single microdisplay,display or projector can also refer to microdisplays, displays orprojectors where a separate display or microdisplay panel is used foreach color channel and they are optically combined such as with atrichroic prism, X-cube prism or dichroic filters. In this instance, forexample three LCoS microdisplays may display 8 bit pixels each and thenan optical masking element LCoS, LCD or DLP microdisplay may be used tomodulate the optically combined 24 (8×3) bit pixels again to reach 30bit or more bit depth (color depth) on the final image.

Having at least one high resolution, small optical element and at leastone low resolution, large optical element may serve more purposes inaddition to creating a final variable resolution image, as avariable-resolution screen may also be useful for allowing the user toswitch between these optical elements for controlling the field of viewof the screen. For example, if the variable-resolution screen apparatusis to be used as a wearable display, the different small or large opticscan be switched between to optically adjust the size of the virtualdisplay. Doing so by digital means instead degrades resolution.Furthermore, one of the optical elements may include a user-controlledzoom lens to adjust the size of the virtual display to any desired sizewithin a range.

A scanned pixel strip (one-dimensional array) or scanned pixel strips(group of one-dimensional arrays) are each a specific type ofmicrodisplay, display or projector image source that may be used inembodiments. A scanned pixel strip has been used as an image source inthe Nintendo Virtual Boy video game console and is described in U.S.Pat. No. 5,682,171. Pixels for different color channels in such an imagesource may be arranged on one strip, different strips per color and/orseveral strips to allow a lower scanning angle. The pixels may beemissive (such as LED, micro-OLED, micro-LED), transmissive (such asLCD) or reflective (such as LCoS, ferroelectric LCoS or DLP). The pixelstrip may be scanned with one or more mirror, prism, or other image orbeam steering element. A scanned pixel strip requires less physicalpixels than a microdisplay, display or projector of same resolution andmay have a lower production cost.

As illustrated in FIGS. 25 and 26, an optical masking element may beused to increase the contrast or bit depth (color depth) of the highresolution small image and/or the low resolution large image also inembodiments that do not involve a duplication element. For atransmissive or reflective microdisplay or display image source, theoptical masking element may be positioned in the optical system beforethe image source to minimize the physical space requirements of thesystem and may be optionally slightly offset to an intermediate imageplane to reduce a screen-door effect generated by the optical maskingelement. The optical masking element may display a much lower resolutionand lower bit depth (color depth) copy of the source image.

Projection lenses may be designed to produce intentional chromaticaberration on the projected image in order to correct the chromaticaberration caused by a single element eyepiece lens.

In embodiments with a large beam splitter, as in FIGS. 22, 23, 32A, thebeam splitter 2209 and/or an optional quarter wave plate (not shown)placed between the beam splitter 2209 and the projection screen 2204 maybe slightly tilted, for example to the side of the head-mounted display,to prevent reflection of the projection lens and stray light fromreaching the viewer's eye 2201 through the eyepiece 2202. Tilting thebeam splitter 2209 may require the projection lens and optionally theoptics before it to be tilted accordingly as well.

FIG. 22 illustrates an embodiment using a mirror 2208 and a beamsplitter 2209 for reducing the physical size of the unit, however thereare additional advantages to using a beam splitter. Video projectionbeams require a certain distance to generate a large enough image on aprojection screen, however the available distance between the eyepiece2202 and projection screen is 2204 determined by the focal length of theeyepiece. For wide field of view eyepiece and small screen dimensionsnecessary for a virtual reality headset, the distance may be quiteshort. Short throw or ultrashort throw projection lenses may be toolarge and heavy for a head-mounted display. When using a beam splitter2209 the optical path of the projection beam is folded, thus allowing aprojection lens designed for longer distance to produce the image ofrequired size, while still allowing the eyepiece to be positioned closeto the projection screen.

Unlike in FIG. 22, the beam splitter 2209 may have a different angle toallow the eyepiece and screen to be positioned closer, as long as themirror 2208 or projector 2203 is rotated accordingly as well. Analternative solution may be the use of a rear projection screen, howeverthe projection beam path behind the screen may still need to be foldedwith a mirror to reduce the size of the unit.

A wide projection screen in a head-mounted display is possible not onlyby using a wide angle projection lens but also by using two or moreprojection lenses with each projecting only part of a horizontal sectionof an image and then optically and/or digitally blending the overlappingedges of the projected images. There is a similar technique used withvideo projectors for blending projection images projected next to eachother known as “edge blending”, although it involves using two separatevideo projectors. Two projection lenses with each only projecting ahorizontal portion the image may be achieved by splitting the image beamwith, but not limited to, a half-silvered mirror beam splitter, ahalf-silvered mirror beam splitter and stencil, a reflective polarizerbeam splitter or a reflective polarizer beam splitter and stencil. Thetwo projection lenses may have different image offsets.

A projection screen may be curved, including curved biaxially, to matchthe curvature of the eyepiece lens. The projection screen may be a rearprojection screen in front of or part of an LCD, OLED, micro-LED orother emissive or transmissive display panel. Such a display panel mayserve as the large image source by generating the large image or it maybe used with a projector image source projecting onto it to provideincreased contrast or color depth for both the high resolution smallimage and low resolution large image generated by the projector bygenerating a copy of the small image, large image or both. The screenmay scatter the light from the display panel, thus making the visibleimage from both the projector and display panel be on the same focalplane of the eyepiece or eye. Such a projection screen may also be madeof a fiber optic taper or faceplate, which may be curved on the sidefacing the eyepiece to match the field curvature of the eyepiece.

A transmissive LCD panel may be placed in front of, behind or inside aneyepiece lens, or in front of or behind a large display panel orrear-projection screen for improving contrast or bit depth (colordepth). The eyepiece lens may be a liquid crystal lens, for example asdescribed in U.S. Pat. No. 10,379,419, or an extra liquid crystal lensmay be added to the head-mounted display either before or after theeyepiece lens to provide variable image focus based on eye trackingdata.

An image may further be phase modulated by a liquid crystal display,liquid crystal microdisplay or liquid crystal shutter array before theeyepiece, inside the eyepiece, after the eyepiece or before the eye ofthe viewer to provide variable focus across the final image without theneed of eye tracking data.

The small image, large image or both may be a light field image toprovide correct accommodation and vergence to the viewer. Light fieldshave higher processing requirements to provide a similar quality imagein terms of resolution and bit depth (color depth), so eye tracking andfoveated content generation may still be performed. Such a light fieldimage may be generated with a lens array or other duplication elementbefore or after a microdisplay, display or projector, as has beendemonstrated by Nvidia with their near-eye light field display which hasa lens array in front of a micro-OLED microdisplay, similarly to howlight field cameras have a lens array in front of an image sensor. Withthis approach the final image resolution is quite low, however with thehigh resolution small image and the low resolution large image it may beincreased.

Alternatively the small image may be a light field image while the largeimage is not, or vice versa. Additionally an electrically tunable lensin the small image or large image optics may be used to adjust the focusof the non-light field high resolution small image based on eye trackingdata or the low resolution large image, thus providing correctaccommodation for a non-light field high resolution small image or largeimage. When the small and large images are consecutive frames, theelectrically tunable lens may also be a liquid crystal lens before,inside or after the eyepiece, or be the eyepiece itself and it may beset to switch based on the current frame.

The large image may be a non-light field image by being outside of thecenter of the viewer's field of view and therefore always out of focus.Light fields may also be generated with a microdisplay with high refreshrate without use of lens arrays as has been demonstrated, for example,by the Institute for Creative Technologies (ICT) of the University ofSouthern California.

A masking or steering element may be used for an illumination beam of afast refresh rate reflective or transmissive microdisplay to control theangle of the light field image beams for each consecutive frame. After alight field image or beam is generated by a microdisplay, display orprojector it may be used to produce a final image for a variableresolution screen the same way a non-light field image or beam may beused, as illustrated in FIGS. 2, 3, 4A, 4B, 5A, 5B, 5C, 5D, 5E, 5F, 5G,511, 13, 14, 17C, 17D, 17E, 17F, 17G, 20, 21, 22, 23, 32A, 32B, 32C.Unlike in some of these figures, an ordinary projection screen may notbe usable in such embodiments, as the screen may diffuse the light. Insome embodiments, a mirror projection screen, mirror or curved mirrormay be used rather than an ordinary projection screen, or the screen maybe the viewer's retina.

Non-Mechanical Beam Steering and Shifting Elements

Instead of mechanical mirrors or optical slabs or non-mechanical imageduplication and masking elements, one or more non-mechanical liquidprism, liquid crystal prism, liquid lens, liquid crystal lens, liquidcrystal microprism array, liquid crystal polarization rotator,birefringent element such as calcite, birefringent element such ascalcite with liquid crystal polarization rotator, electrically ormagnetically controllable birefringent element, and/or electrically ormagnetically controllable birefringent element with liquid crystalpolarization rotator may be used. A liquid crystal prism may refer to apolarization or diffraction grating with a liquid crystal polarizationrotator and/or a spatial light modulator such as a transmissive LCDmicrodisplay functioning as polarization or diffraction grating with orwithout a liquid crystal polarization rotator. These elements may bestacked to increase possible beam steering angles or beam shiftingranges. The incoming beam may be a polarized beam in embodiments.Dispersion introduced by these elements may be corrected digitally.

Positioning the high resolution small image on the final image in a widerange of positions with these non-mechanical elements is possible whenused with a projector image source as the system, and the image beam isrelatively small before and inside the projection lens and requiresrelatively small steering angles and shifting distances as well.

FIGS. 27A-27D illustrate how non-mechanical elements may shift an imagebeam and position an image, in accordance with an embodiment. FIGS.27A-27D illustrate that the distance between the positions of the smallimage, which may be positioned by non-mechanical elements, may be aslong as the size of the small image in each axis or may be shorter thanthe size of the small image in one or more axis. FIGS. 27A-27C furtherillustrate how much smaller the viewable area of the small image may befor the viewer to not experience a noticeable visual jump when switchingbetween two or more positions. FIGS. 27A-27C further illustrate anamount of viewable area for nine positions of the small image.

FIG. 27A shows two possible image positions marked with black circles2701 with a distance between each other matching the size of the imagein each axis, with the sizes of the image when positioned illustrated bydotted line squares 2702. This kind of positioning may be impractical asthe viewer will notice a jump when the position is changed between thetwo illustrated positions. Most of the listed non-mechanical beamsteering and shifting elements cannot position the image in an arbitraryposition between the two illustrated positions in FIG. 27A in each frameand can only position the image in two positions. The elements may bestacked to provide more possible positions. However, there is a limit tohow many elements may fit in a small system. There is a solution to thisproblem which involves using a smaller visible portion of the wholeimage with such positioning.

FIG. 27B shows a different distance between two possible positions,where the distance between the two positions is smaller than the size ofthe image in each axis. FIG. 27C illustrates what size the visibleportion of the image may be, the visible portion illustrated as a solidsquare 2704, for the desired result. In FIG. 27C the visible portion issmall enough that it may be displayed both when the whole image ispositioned on the left position as well as the right position, as thesize of the image in each axis is larger than the distance between thesepositions. In an example where the visible portion has to smoothly movefrom a leftmost position to a rightmost position possible by two suchpositions, the image may first be positioned on the left position andthe visible portion may be digitally positioned on the image to occupythe leftmost part of the image. Then the positioning of the visibleportion may be performed digitally until the visible portion reaches alocation illustrated in FIG. 27C which is either the rightmost part ofthe image when the image is positioned to the left position or theleftmost part of the image when the image is positioned to the rightposition. To move the visible portion of the image 2704 even more rightthe image may be positioned to the right position and the visibleportion may immediately be shifted to cover the leftmost portion of theimage rather than the rightmost, and from there on the positioning ofthe visible portion will again only be performed digitally bypositioning the visible portion on the image until the visible portionreaches the rightmost part of the image.

FIG. 27D shows 9 such positions 2705 of the image which illustrates howmuch area the visible portion of the image may be positioned in, where asingle dotted rectangle from 2701 may be used as reference for the sizeof the area. As may be seen, while the visible portion is smaller thanthe image itself, the area it may be positioned in is larger. Nine suchpositions may be achieved with, for example, 6 liquid crystal prisms (3for each axis) or 4 birefringent elements such as calcite with liquidcrystal polarization rotators (2 for each axis), where each element isconfigured to shift a beam in only one axis. Such a positioning methodmay also be used for duplication and masking elements where theduplicate small images partially overlap on an intermediate image planeor on the final image.

FIGS. 28A-28C illustrate various non-mechanical beam steering elementsand how the beam may be shifted with each, in accordance with anembodiment. FIG. 28A illustrates 3 possible parallel beam paths of abeam going through 3 liquid crystal prisms. FIG. 28B illustrates 3possible parallel beam paths of a beam going through 2 birefringentelements such as calcite and 2 liquid crystal polarization rotators.FIG. 28C illustrates 2 possible parallel beam paths of a beam goingthrough 2 birefringent elements such as Rochon prisms and 2 liquidcrystal polarization rotators.

In FIG. 28A three liquid crystal prisms 2802, 2803, 2804 areillustrated, which may be set to steer the incoming beam 2801 by aspecific angle or let the beam pass through. The first liquid crystalprism 2802 is able to steer the incoming beam by a specific angle whilethe other two 2803, 2804 may steer the beam coming from first prism 2802in the opposite angle, thus causing the beam leaving them 2805, 2806,2807 to be parallel to the incoming beam 2801, but optionally shifted.Such liquid crystal prisms have two or three electrically adjustablestates. In this example two states are assumed and the first state letsthe beam pass through without being steered while the second statesteers the beam. The liquid crystal prisms may also be not electricallytunable and let the beam pass through or steer it based on thepolarization state of the beam, in which case a liquid crystalpolarization rotator or other electrically tunable polarization rotatormay be used between the liquid crystal prisms (not illustrated) forcontrolling the polarization state of the beam.

In FIG. 28B birefringent elements 2810, 2812 such as calcite with liquidcrystal polarization rotators 2809, 2811 may shift the incoming beam2808 by a specific amount or let the beam pass through depending on thestate of the liquid crystal polarization rotators 2809, 2811 beforethem, with the beam leaving them 2813, 2814, 2815 being parallel to theincoming beam 2808 angle, but optionally shifted.

In FIG. 28C a specific type of birefringent elements, known as Rochon orSenarmont prisms 2818, 2820, with liquid crystal polarization rotators2817, 2820 are shown. Due to the optical function of these prisms, onlytwo possible beam paths 2821, 2822 parallel to the incoming beam 2816may be achieved with two of such prisms.

The above figures illustrate the beam shifting only in one axis, but theother axis may be shifted as well, for example by stacking more of suchelements one after the other.

More Duplication Elements

While a lens array may the most straightforward duplication element forcreating duplicated beams with minimal amount of optical aberrations andrequired physical space, there are also other optical elements that mayserve as a duplication element, such as, but not limited to, a pinholearray, diffractive optical element (“DOE”), holographic optical element(“HOE”), optical element based on fiber optics such as fiber opticstaper or fiber optic faceplate, array of beam splitters, array of beamsplitters combined with lens arrays, freeform prisms combined with lensarrays or a single beam splitter, PBS cube or wire-grid polarizer X-Cubeprism combined with lens arrays. A pinhole array is a simple alternativeto a lens array but may suffer from much lower light throughput andworse optical aberrations. A diffractive optical element or diffractiongrating may duplicate images. A disadvantage with DOEs may be that theywork well with monochromatic laser light, so may only be practical whenused with separate microdisplays for each color channel and insituations where there is enough physical space. A crossed fiber gratingis a specific type of diffraction grating that may be used. Aholographic optical element may be used, which also works by diffractionand may have similar disadvantages as DOEs.

FIG. 29 illustrates a fiber optic taper or faceplate from four differentangles, with each optical fiber of the taper or faceplate branching outinto 4 fibers and thus each pixel going through 4 fibers and producing 4duplicate pixels, in accordance with an embodiment. A fiber optic taperor faceplate may function as a duplication element by having eachoptical fiber branch out into two or more fibers, and thus each pixelgoing through one or several fibers may produce two or more duplicatepixels, as illustrated in FIG. 29, where the fibers are illustrated as 4extruded squares 1, 2, 3, 4. Depending on the size of the fiber optictaper or faceplate face on the opposite side, the optical maskingelement, for example a transmissive LCD display panel, may be placedright in front of the faceplate or taper face.

An array of beam splitters may be used to split the beam into two ormore duplicate beams. The beam splitters may be half-silvered mirrorbeam splitters or reflective polarizer beam splitters. In place of beamsplitters, a freeform prism may alternatively be used. An extra lensarray after the freeform prism may be used to undo any geometricdistortion or other optical aberrations imposed on the image by thefreeform prism.

When using a beam splitter or beam splitter array (stacked or a group ofbeam splitters) as a duplication element by itself, beams coming out ofsuch a beam splitter or beam splitter array may have different lengthoptical paths that should be accounted for, for example by using a lensor lens array after or before each beam splitter where each lenslet onthe lens array meant for a specific duplicate beam may have a differentlens profile. The lens array may have more than one lens element forcorrecting optical aberrations. Alternatively, a fiber optic faceplateor taper may be placed after each beam splitter with each having adifferent length.

A single beam splitter, PBS cube or wire-grid polarizer X-Cube,optionally combined with lenses and/or lens arrays after it, may makesense where the small image covers a significant field of view of theviewer. Such a configuration also makes sense where, for example, thefield of view should only be doubled horizontally. Although for LCoSillumination rather than imaging, such beam splitter diagrams areprovided in U.S. patent 20130063671A1.

With an image source with a fast refresh rate such as OLED, micro-OLED,micro-LED or ferroelectric LCoS (FLCoS) microdisplay, it is possible touse a very low resolution but high refresh rate optical masking element,such as Pi-cell shutter array or ferroelectric liquid crystal shutterarray, with the resolution of the optical masking element, such as theamount of optical shutters, matching the amount of duplicate beamsproduced by the duplication element.

FIG. 30 illustrates how a fast refresh rate image source may be usedwith a low resolution high refresh rate optical masking element forpositioning of the small image on the final image. In FIG. 30 there is asource image 3001, with portions 1, 2, 3 and 4. Frames 3002, 3003, 3004,3005 are consecutive frames displayed by the fast refresh rate imagesource. For the first consecutive frame 3002, only the bottom-rightshutter of the optical masking element is set to transmit the beam. Forthe consecutive frame 3003 only the bottom-left shutter of the opticalmasking element is set to transmit the beam. For the consecutive frame3004, only the top-right shutter of the optical masking element is setto transmit the beam. For the consecutive frame 3005, only the top-leftshutter of the optical masking element is set to transmit the beam.Persistence of vision blends the 4 consecutive frames 3002, 3003, 3004,3005 into a single optically and digitally reconstructed small image3006. A different configuration of displayed source image portion andshutter visibility in each consecutive frame is also possible.

If the optical masking element is larger than 2×2, then a 2×2 portion ofthe optical masking element may be used in the consecutive framesdescribed above and the shutters outside of that portion may remainnon-transmitting in those consecutive frames. A 5×5 or 3×3 or othersmall size optical shutter array optical masking element may be a morecost effective solution than using a high resolution reflective ortransmissive display or microdisplay for the optical masking element.Another advantage with this approach is the ability to have the opticalmasking element not on an image plane 1505 but also right before orright after the duplication element 1503, thus allowing to minimize thephysical space requirements of the system. With such a fast refresh rateimage source and low resolution high refresh rate optical maskingelement, the positioning precision is only limited by the resolution ofthe image source. There may be no need for an intermediate image plane1505 with these embodiments and the image may be formed directly on thescreen.

Instead of blocking all but one duplicate beam in each consecutiveframe, it is possible to direct the source image beam to only onedirection at a time with, for example, switchable liquid crystalpolarization rotators and reflective polarizer beam splitters, forincreasing light efficiency. A disadvantage with this approach may bethe physical space required.

With any duplication element, an extra LCD shutter or polarizer in frontof or after each duplication element may be used for reducing straylight.

Increasing Resolution with Intentional Image Distortion

Fixed foveation or static foveation are terms that are used to refer toan image with higher pixel density in its middle portion and field ofview which is not repositioned based on the viewer's gaze direction. Insome VR headsets this is performed on the image digitally to reduce CPUand GPU usage. With fixed foveation the resolution in any part of theimage cannot be higher than the pixel density the screen allows. In someVR headsets, fixed foveation is also a result of using a single elementlens which produces pincushion distortion and more pixels getconcentrated on the middle of the final image. The amount of distortionfrom single element lenses is not high and is not considered asignificant increase to the resolution of the middle portion of thefinal image.

It is possible to have actual significant variable resolution whileusing an ordinary eyepiece lens or even other optical elements whenusing projection lenses and screens. The solution involves applyingintentional image distortion to the image or beam before it reaches theprojection screen or on the projection screen, such as inside theprojection lens or before the projection lens. In one embodiment, imagedistortion is applied inside the small or large image optical element.Thus the image that is projected on the projection screen already hasthe image distortion applied, and the distortion does not depend on theeyepiece which is facing the projection screen. This allows the use ofan ordinary eyepiece lens with the screen. This is another advantagewith using projection beams and projection screens rather than displaypanels.

FIG. 31 illustrates various methods of increasing the resolution of animage, in accordance with an embodiment. In FIG. 31, intentional imagedistortion 3103 is illustrated. Intentional image distortion may also becombined with a high resolution small image and a low resolution largeimage. One purpose may be to ensure somewhat high resolution when theeye tracking is turned off, when eye tracking malfunctions or when theeye tracking falls behind due to a lower refresh rate than the eyesaccadic movement. In embodiments where the position of the highresolution small image on the low resolution large image is not changedand is in the middle portion of the low resolution large image,pincushion or similar distortion may be applied to the high resolutionsmall image while barrel or similar distortion is applied to the lowresolution large image, to ensure the pixels which are in the middle ofthe low resolution large image are not wasted by being optically ordigitally masked and rather higher pixel density is provided to thevisible portion of the low resolution large image as well as to the highresolution small image.

In embodiments with a rear projection screen or a separate LCD, OLED ormicro-LED low resolution large image source, the distortion on the lowresolution large image may be achieved with a fiber optic faceplate ortaper adhered or attached to the rear projection screen or lowresolution large image source where the size and/or arrangement of thefiber tips on one surface of the faceplate or taper do not match withthe other surface.

Increasing Resolution with Pixel Shifting

Pixel shifting or beam shifting refers to a technique used in digitalcameras and video projectors for increasing the capture or displayresolution by optically shifting or offsetting the capture or projectedbeam after each consecutive frame. With digital cameras the sensorpixels may be physically shifted and a high resolution image generateddigitally by digitally combining the offset frames while with videoprojectors an actuator with a rotating/tilting optical slab in front ofa DLP or LCoS microdisplay is employed, such as Optotune© XPR-9-2P, andeach frame the microdisplay generates several consecutive frames whichthe actuator slab shifts to have pixels fill the pixel gaps of thepixels from the previous position. Persistence of vision blends theconsecutive frames into a single higher resolution frame. The actuatormay shift 2 to 4 or more consecutive frames for each complete frame toincrease the resolution 2 to 4 or more times. This technique is how manyvideo projectors achieve 3840×2160 pixels resolution from a microdisplaywith only 1920×1080 pixels resolution. The technique helps keepmicrodisplays small, and helps meet higher resolution requirementswithout reaching the diffraction limit in the optical system. Theactuator may also be used to only shift the same frame pixels justenough to fill the pixel gaps and reduce the screen-door effect withoutincreasing the resolution.

With LCoS, pixel shifting may not be practical due to the relativelyslower liquid crystal pixel switching speeds of LCoS and sinceconsecutive frames are already used to display separate color channelsof a frame with “color-sequential” operation which is the method usedfor AR. Color-sequential operation suffers from an image artifact knownas the “rainbow artifact”. As described already the rainbow artifact maybe eliminated by using a separate LCoS microdisplay per color channeland combining the different color channel beams with dichroic filters,an X-Cube prism and/or a trichoric prism. When doing this the refreshrate of the LCoS microdisplay is increased by 3, 4 or more times sincethe same LCoS microdisplay does not need to display the different colorchannels of each frame. The frame rate may then be used for displayingmore consecutive frames together with pixel shifting to increase theresolution of the final image.

FIG. 31 illustrates various methods of increasing the resolution of animage, in accordance with an embodiment. FIG. 31 shows how a pixel gridlooks 3102 before and after pixel shifting, in accordance with anembodiment.

Pixel shifting may also be performed by mirrors attached to actuators.Pixel shifting may also be achieved without mechanically moving parts,such as with a liquid or liquid crystal prism or microprism array orliquid crystal polarization rotator before or after a birefringentoptical element such as a calcite or quartz plate. With a non-mechanicalcomponent pixel shifting may also be performed for a larger LCD, OLED ormicro-LED display panel and may also be placed before, inside or afteran eyepiece, although the pixel shifting optical element may be fragilein such an embodiment.

With video projectors the 4 consecutive frames are generated from onehigher resolution frame, however with VR headsets the frame is updatedbased on head rotation and position so rendering all of the 2, 3, 4 ormore consecutive frames in advance may result in visual artifacts. Asolution is not shifting 2, 3, 4 or more consecutive frames of a singlehigh resolution frame rendered in advance but having the consecutiveframes rendered in real time and/or the consecutive frames digitallywarped in real-time for the correct pixel data to match the new headrotation and position.

Besides pixel shifting per consecutive frame, pixel shifting may also beperformed per frame, for example for the low resolution large image,provided the refresh rate is fast enough to not introduce noticeableflicker.

A pixel shifting pattern may be intentionally not synced for the imagesfor each eye to reduce possible noticeable flicker.

Pixel shifting has some similarities to interlaced video and displaying,a difference being that the refresh rate is higher. Additionally, sincethere isn't a complete black pixel row displayed for each frame withpixel shifting but rather the pixel rows of the consecutive framespartially overlap, and since the consecutive frames may be rendered inreal time rather than in advance, visual artifacts such as “combing” areless noticeable. Additionally, due to the real time nature of VR and ARcontent, pixel shifting may be turned off during rapid head or eyemovement or turned off for objects displayed in the frames with rapidmotion, by not rendering pixel information for them in a consecutiveframe, to reduce any possible remaining visual artifacts.

Increasing Resolution by Changing the Aspect Ratio of the Source Image

A typical aspect ratio of a VR head-mounted display image is about 1:1per eye. Using a 16:9 aspect ratio microdisplay as an image source forsuch a VR headset may result in very underutilized pixel count withnearly half of the pixels unused. However with a projector 2003, 2103,2203 inside a VR headset there may be enough physical space forrelatively small projection lenses. By using cylindrical, freeform oranamorphic lens elements or anamorphic prisms it is possible to changethe aspect ratio of the source image from 16:9 to something like 1:1.While pixels may not be near-perfect squares anymore, at certainmagnification and pixel density it may not be perceivable and mayprovide almost twice the pixel density in one axis of the image. FIG. 31illustrates how pixels of such source image which has had its aspectratio changed look like 3101.

Combining Intentional Image Distortion, Pixel Shifting and Changing theAspect Ratio of the Source Image

As illustrated in FIG. 31, intentional image distortion, pixel shiftingand changing the aspect ratio of the source image may be combined 3104to increase the resolution of the image even further. For example, ifintentional image distortion 3103 is applied to a source microdisplaywith a resolution of 1920×1080 pixels to produce 1.33 times more pixelsin the middle portion of the image, then the resulting image may have aresolution of around a 2553×1436 pixel resolution image when viewing themiddle portion of the image and around 1440×812 pixel resolution imageon the very edges. If a two consecutive frame pixel shifting 3102 isapplied to such an image before or after the intentional imagedistortion 3103, the resulting image may have a resolution of around a2553×2872 pixel resolution image when viewing the middle portion of theimage and around 1440×1624 pixel resolution image on the very edges. Ifthe aspect ratio of the source image is changed from 16:9 to closer to a1:1 ratio 3101, then a similar pixel density in both axes of the finalimage may be achieved with an eyepiece with equal or almost equalhorizontal and vertical fields of view. This may be applied to both thehigh resolution small image and low resolution large image.

As an example, if the image source is made of three LCoS panels with a240 Hz refresh rate combined with an X-Cube prism or dichroic prisms,two consecutive frames may be used for displaying the two pixel-shiftedportions of the high resolution small image and the next consecutive 2frames may be used for displaying the two pixel-shifted portions of thelow resolution large image. Such high resolution images may requireconsiderable amount of processing power to render so eye tracking andfoveated content generated based on the eye tracking data may still beused.

One or more of these methods may be used with a lens array imageduplication element to allow a reduction in the size of the lens arrayand avoid issues that may arise with lens arrays with many lenslets,such as light efficiency and diffraction limit. One or more of thesetechniques may be applied to both the small image and large image, andthey may share the optical elements performing these tasks, for exampleby having the lens elements before, inside or part of the projectionlens. Although these three methods may not increase resolution as muchas having a separate small and large image, they may help in increasingthe size of the high resolution small image and/or the size of thetransition region between the small image and large image and may allowthe low resolution large image to have more resolution, all of which mayprovide a more seamless variable resolution image to the viewer.

High Gain Video Projection Screen for Head-Mounted Display

Video projection screen gain refers to the brightness of the image fromthe screen, usually using a barium sulfate or magnesium carbonatesurface as a reference which have a gain of 1.0. A video projectionscreen works by reflecting and scattering the image beam projected ontoit in a specific way. High gain usually corresponds with a narrowscattering angle and therefore limited viewable area in front of orbehind the screen. Unlike traditional video projectors, VR or ARhead-mounted displays have screen and projected image positions mostlyfixed to the viewer's eye, so the projection screen may be engineered tohave much smaller viewable area corresponding to the size of the eye-boxand therefore provide much higher gain than a traditional videoprojection screen. On the other hand, there are stricter requirementsfor the screen surface and structures such as microscopic glass beads onthe screen since the video projection pixels inside a head-mounteddisplay are much smaller as well and parts of such structures may end upeither outside of the depth of focus of the projection or degraderesolution by causing too much scattering or cause diffraction. Asolution to this issue may be to have the projection screen inside ahead-mounted display be used to not only scatter the projected pixelbeams but to also steer them by having the screen be biaxially curved,as is illustrated in FIG. 32A. For a rear projection screen, a curvedmirror or lens may be used for steering instead and the rear projectionscreen itself may be flat and responsible for scattering but notsteering, as is illustrated in FIGS. 32B and 32C, or the screen maystill be biaxially curved for matching a field curvature of an eyepiecelens. The screen may then be engineered to increase the reflected pixelbeam angle due to scattering by a specific amount required by theeyepiece eye-box and not more for providing a very high gain projectionscreen.

FIG. 32A illustrates how a curved projection screen may be used withengineered reflective and scattering properties to both steer theprojection beam as well as scatter it by a specific amount to achieve avery high gain projection screen, in accordance with an embodiment. InFIG. 32A the projector 3203 produces an image that is reflected off of amirror 3207. The mirror 3207 reflects the light to a beam splitter 3208which reflects the projection beam 3205 onto the curved projectionscreen 3204. The curved projection screen 3204 both steers and scattersthe projection beam 3205. The human eye 3201 is behind an eyepiece lens3202 or other optics and the viewer sees through the beam splitter 3208the steered and scattered light 3206 from the image on the screen 3204.

FIG. 32B illustrates how a curved mirror, beam splitter and rearprojection screen with engineered scattering properties may be used tosteer the projection beam by the curved mirror and scatter it by aspecific amount by the rear projection screen to achieve a very highgain projection screen, in accordance with an embodiment. In FIG. 32Bthe projector 3213 produces an image beam that is reflected off of amirror 3217. The mirror 3217 reflects the light to a beam splitter 3218which reflects the projection beam 3215 onto the curved mirror 3214. Thecurved mirror 3214 only steers and does not scatter the projection beam3215. The projection beam 3215 is then steered and focused onto a rearprojection screen 3219, which scatters but does not steer the projectionbeam 3215. The human eye 3211 is behind an eyepiece lens 3212 or otheroptics and the viewer sees the scattered light 3216 from the image onthe screen 3219. The curvature of the mirror 3224 may also be freeform,for example for the mirror to steer a projection beam which may not beperfectly perpendicular to it and may not be reflected from a beamsplitter.

FIG. 32C illustrates a different configuration of how a curved mirror,beam splitter and rear projection screen with engineered scatteringproperties may be used to steer the projection beam by the curved mirrorand scatter it by a specific amount by the rear projection screen toachieve a very high gain projection screen, in accordance with anembodiment. In FIG. 32C the projector 3223 produces an image beam thatis reflected off of a mirror 3227. The mirror 3227 reflects the light toa beam splitter 3228 which on the first pass transmits the projectionbeam 3225 onto the curved mirror 3224. The curved mirror 3224 onlysteers and does not scatter the projection beam 3225 onto a rearprojection screen 3229 through a reflection from the beam splitter 3228.The human eye 3221 is behind an eyepiece lens 3222 or other optics andthe viewer sees the scattered light 3226 from the image on the screen3229.

To reduce light loss and image artifacts, the beam splitter 3208, 3218,3228 may be a reflective polarizer beam splitter and there may be anadditional quarter wave plate placed between the reflective polarizerbeam splitter 3208, 3218, 3228 and the curved screen 3204 or mirror3214, 3224. An additional absorptive polarizer may be placed before theeye or before, inside or after the eyepiece to filter out anyreflections or stray light from the projection lens on the reflectivepolarizer or quarter wave plate.

Size of projected pixels is less of an issue with HMPDs (Head-MountedProjective Display), and a regular retro-reflective projection screenmay be used with them, which are known to provide considerable amount ofscreen gain.

Contrast and Bit Depth (Color Depth) Enhancement without Optical MaskingElement

Consecutive frames or time multiplexed as well as optically overlayedsmall and large images may be used not only to increase resolution, butalso to increase contrast and bit depth (color depth) of the finalimage. With pixel shifting, consecutive frames, since their pixelspartially overlap, may be used for enhancing contrast or bit depth(color depth) of the final image. With bit depth (color depth)enhancement specifically, one consecutive frame may be used to display aspecific color information, while the other may display the rest of thecolor information. Brightness of the consecutive frames may be adjustedelectrically or optically, if needed.

FIG. 33 illustrates how the pixel information displayed by the highresolution small image and the region of the low resolution large imagethat corresponds to the high resolution small image may be used togetherto increase at least one of a contrast or bit depth (color depth) of thesmall image portion of the final image, in accordance with anembodiment. With embodiments with a small image and large image asillustrated in FIG. 33, instead of completely masking the portion of thelow resolution large image that corresponds to the high resolution smallimage (referred to as a high resolution small image copy portion 3303 ofthe large image 3302) by displaying black pixels there, a copy of thehigh resolution small image may be displayed there instead 3303, albeitat a lower resolution. The brightness may be electrically or opticallyadjusted for the small image 3301 and digitally for the copy of the highresolution small image portion 3303 on the large image 3302 to ensurebrightness uniformity across the final image 3304. Since the brightnessof the high resolution small image 3301 is adjusted electrically oroptically, the pixel information displayed by the high resolution smallimage 3301 and the high resolution small image copy portion of the largeimage 3303 may serve to increase at least one of a contrast or bit depth(color depth) of the small image portion of the final image.

Enhancing Non-VR Content Displayed by a Variable Resolution Screen

A variable resolution screen head-mounted display may be used to displaynon-VR content, for example a desktop of an operating system. There areseveral visual effects to enhance such 2D frames with a variableresolution screen head-mounted display not possible with other displaydevices and to make the 2D frames appear to cover more field of viewthan they actually do.

FIG. 34 illustrates various visual effects that may be applied beyond asource 2D frame on a variable resolution screen, in accordance with anembodiment. In FIG. 34, image 3401 shows a source 2D frame while images3402, 3403 and 3404 show visual effects applied beyond the frame. Image3402 shows a visual effect where the pixels on the edges of the frameare used beyond the 2D frame. Various image filters may be applied tothese pixels, for example a gaussian blur or zoom blur (what isillustrated) to achieve various effects. Similar video effects exist onsome TVs where there are colored LEDs on the back side of the TVilluminating the wall behind the TV. Since here there is more control onwhat can be displayed beyond the edges of the frame rather than a simplediffuse reflection as in the TV example, more effects are possible.Image 3403 illustrates such an advanced effect. The source frame isdigitally scaled up, placed on the back of the source frame and blurredto not take the attention of the viewer from the source small framedisplayed in the middle. A similar effect is commonly used in videoswhen a non 16:9 aspect ratio content is displayed on a 16:9 frame,instead of merely displaying black or white borders. In our example theeffect is not only applied to two edges of the 2D frame but all aroundit. Image 3404 illustrates a more advanced effect which needs moreinformation from the source rather than just the 2d frames. In themiddle the actual 2d frame is displayed, while beyond the frame a lowerresolution continuation of the frame is displayed. This effects requiresthe video input source to supply the video beyond the frame as well. Thesource in this instance may be a PC program or video game.

The head-mounted display with a variable resolution screen may take the2D frame from an input, for example a desktop of an operating system,while displaying another content on the pixels beyond the frame, forexample a 3D rendering of a virtual office or home environment. In thisexample the 2D frame may appear floating in the middle of the viewer'sfield of view while the 3D rendering may be updated to match theviewer's head position and rotation. Such a visual effect may be usefulwhen switching between a desktop of an operating system and a VR programwithout having to take off the head-mounted display to be able to seethe desktop on a monitor and then put it back on again.

The foregoing devices and operations, including their implementation,will be familiar to, and understood by, those having ordinary skill inthe art. All sizes and proportions used in this description could bescaled up or down or changed without impacting the scope of theseinventions.

The above description of the embodiments, alternative embodiments, andspecific examples, are given by way of illustration and should not beviewed as limiting. Further, many changes and modifications within thescope of the present embodiments may be made without departing from thespirit thereof, and the present invention includes such changes andmodifications.

What is claimed is:
 1. A head-mounted display, comprising: an imagesource configured to output one or more image components; and one ormore optical element configured to receive the one or more imagecomponents and output one or more images focused onto a projectionscreen that scatters and steers the one or more images for viewing by aviewer's retina.
 2. The head-mounted display of claim 1, wherein theimage source comprises a separate display or microdisplay panel for eachcolor channel of a plurality of color channels, the head-mounted displayfurther comprising: at least one of a trichroic prism, an X-cube prismor a dichroic filter to optically combine the plurality of colorchannels.
 3. The head-mounted display of claim 2, wherein thehead-mounted display comprises the projection screen, and wherein theprojection screen is a rear projection screen in front of or part of anemissive or transmissive display panel, and wherein the projectionscreen is configured to scatter light from the emissive or transmissivedisplay panel to cause a visible image from the projection screen andthe emissive or transmissive display panel to be on a same focal plane.4. The head-mounted display of claim 3, wherein the emissive ortransmissive display panel is a liquid crystal display (LCD) panel, anorganic light emitting diode (OLED) display panel, or a micro-LEDdisplay panel.
 5. The head-mounted display of claim 2, furthercomprising: an emissive or transmissive display panel configured toprovide increased contrast or color depth for the one or more images;and the projection screen, wherein the projection screen is a rearprojection screen in front of or part of the emissive or transmissivedisplay panel.
 6. The head-mounted display of claim 5, wherein theemissive or transmissive display panel is a liquid crystal display (LCD)panel, an organic light emitting diode (OLED) display panel, or amicro-LED display panel.
 7. The head-mounted display of claim 1, whereinthe projection screen is a rear projection screen.
 8. The head-mounteddisplay of claim 1, wherein the projection screen is biaxially curved.9. The head-mounted display of claim 1, further comprising: a liquidcrystal display, a liquid crystal microdisplay, or a liquid crystalshutter array configured to phase modulate at least one image of the oneor more images.
 10. The head-mounted display of claim 1, furthercomprising: a pixel shifting element configured to perform pixelshifting to increase a resolution of at least one image of the one ormore images.
 11. The head-mounted display of claim 10, wherein the pixelshifting element is selected from the group consisting of: a liquidprism, a liquid crystal prism, a liquid crystal microprism array, and aliquid crystal polarization rotator and a birefringent optical elementbefore or after the liquid crystal polarization rotator.
 12. Thehead-mounted display of claim 1, wherein the projection screen is afront projection screen that is biaxially curved and that both steersand scatters a projection beam comprising the one or more images. 13.The head-mounted display of claim 1, wherein the one or more imagecomponents comprises a plurality of consecutive frames, and wherein thehead-mounted display is to use the plurality of consecutive frames toenhance a contrast or a bit depth of the one or more images.
 14. Ahead-mounted display, comprising: an image source configured to outputone or more image components; and one or more optical element configuredto receive the one or more image components and output one or moreimages onto a projection screen; wherein the one or more imagecomponents comprise a high resolution, small image component and a lowresolution, large image component, wherein the one or more opticalelement is configured to receive the high resolution, small imagecomponent and output a high resolution, small image and to receive thelow resolution, large image component and output a low resolution, largeimage, and wherein the high resolution, small image and the lowresolution, large image appear as a variable resolution image on theprojection screen.
 15. The head-mounted display of claim 14, wherein theone or more optical element comprises a small image optical elementconfigured to receive the high resolution, small image component and alarge image optical element configured to receive the low resolution,large image component, and wherein the small image optical element andthe large image optical element share one or more of their constituents.16. The head-mounted display of claim 14, wherein the one or moreoptical element comprises a small image optical element configured toreceive the high resolution, small image component and a large imageoptical element configured to receive the low resolution, large imagecomponent, and wherein the small image optical element includes aduplication element and an optical masking element, and wherein: theduplication element is to receive the high resolution, small image andoutput a plurality of duplicates of the high resolution, small image;and the optical masking element is to mask off one or more of theplurality of duplicates of the high resolution, small image such that atleast portions of one or more duplicates of the high resolution, smallimage remain, wherein at least the portions of the one or moreduplicates of the high resolution, small image form a complete singleduplicate of the high resolution, small image that is focused onto atarget position on the projection screen.
 17. The head-mounted displayof claim 16, wherein the duplication element comprises: a beam splitter;and a lens or a lens array.
 18. The head-mounted display of claim 16,wherein the duplication element comprises: a beam splitter array; and alens or a lens array.
 19. The head-mounted display of claim 16, whereinthe duplication element comprises: at least one of a multi-element lensor a lens array configured to reduce optical aberrations in theplurality of duplicates of the high resolution, small image.
 20. Thehead-mounted display of claim 14, further comprising: an image steeringelement configured to control a placement of the high resolution, smallimage on the projection screen.
 21. The head-mounted display of claim20, wherein the image steering element comprises a liquid crystal prism.22. The head-mounted display of claim 20, wherein the image steeringelement comprises a birefringent element and liquid crystal polarizationrotator.
 23. The head-mounted display of claim 14, wherein at least oneof the high resolution, small image or the low resolution, large imageis a light field image.
 24. The head-mounted display of claim 14,wherein pixel information displayed by the high resolution, small imageand a region of the low resolution, large image that corresponds to thehigh resolution, small image are used together to enhance a contrast orbit depth of the high resolution, small image as it appears in thevariable resolution image on the projection screen.
 25. A head-mounteddisplay, comprising: an image source configured to output one or moreimage components; one or more optical element configured to receive theone or more image components and output one or more images onto aprojection screen that scatters the one or more images for viewing by aviewer's retina through an eyepiece; and the eyepiece, configured toview the projection screen, wherein the eyepiece comprises a liquidcrystal lens to provide variable image focus based on eye tracking data.26. A head-mounted display, comprising: an image source configured tooutput one or more image components; and one or more optical elementconfigured to receive the one or more image components and output one ormore images having an intentional image distortion onto a projectionscreen, the intentional image distortion comprising at least one ofpincushion distortion or barrel distortion; wherein the head-mounteddisplay is configured to increase a resolution of at least one image ofthe one or more images with the intentional image distortion.
 27. Thehead-mounted display of claim 26, wherein the projection screen is aviewer's retina.
 28. A head-mounted display, comprising: an image sourceconfigured to output one or more image components; one or more opticalelement configured to receive the one or more image components andoutput one or more images onto a projection screen; and an opticalmasking element configured to display a copy of at least one image ofthe one or more images to provide at least one of an increased contrastor an increased color depth for the at least one image, wherein theoptical masking element comprises at least one of a microdisplay or adisplay.
 29. The head-mounted display of claim 28, wherein theprojection screen is a viewer's retina.
 30. A head-mounted display,comprising: an image source configured to output one or more imagecomponents having a first aspect ratio; and one or more optical elementconfigured to receive the one or more image components having the firstaspect ratio and output one or more images having a second aspect ratioonto a projection screen; wherein a resolution of the one or more imagesis increased based on changing from the first aspect ratio to the secondaspect ratio for the one or more images.
 31. The head-mounted display ofclaim 30, wherein the projection screen is a viewer's retina.
 32. Ahead-mounted display, comprising: an image source configured to outputone or more image components, wherein the image source comprises aseparate display or microdisplay panel for each color channel of aplurality of color channels; at least one of a trichroic prism, anX-cube prism or a dichroic filter to optically combine the plurality ofcolor channels; and one or more optical element configured to receivethe one or more image components and output one or more images onto aprojection screen, wherein the head-mounted display is a head-mountedprojective display (HMPD) and the projection screen is an externalprojection screen that is not part of the head-mounted display.